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PREFACE 


w ' 

This  document,  the  Interservice1  Procedures  for  Instructional 

Systems  Development:  Executive  Summary  and  Model  is  a description 

of  the  approved  techniques  and  procedures  to  be  followed  in  the 

development  and  conduct  of  interservice  training.  It  was  prepared 

! ' under  Contract  Number  N-61339-73-C-Q150  between  the  Center  for 

Educational  Technology  at  Florida  State  University  and  the  U.S. 
i 

j Army  Combat  Arms  Training  Board,  Ft.  Benning,  Georgia.  After  the 

i inception  of  the  project,  the  Interservice  Committee  for  Instruc- 

i 

j tional  Systems  Development  became  the  approving  authority  by 

] agreement  with  the  Army.  This  change  in  approving  authority  broad- 

j ' ened  the  project's  scope  to  include  requirements  from  the  Air  Force, 

: Marines,  and  Navy  as  well  as  from  the  Army. 
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^ PHASE  I 

ANALYZE 


PHASE  II 

DESIGN 


PHASE  III 

DEVELOP 


PHASE  IV 

IMPLEMENT 


PHASE  V 

CONTROL 


THE  FIVE  PHASES  ARE: 


Inputs,  piocesses.  and  outputs  in  Phase  I are  aii  based  on  job  information.  An  inventory  of  job 
tasks  is  compiled  and  divided  into  two  groups:  tasks  not  selected  for  instruction  and  tasks 
seine  ted  for  instruction.  Performance  standards  for  tasks  selected  for  instruction  are  determined 
by  interview  or  observation  at  job  sites  and  verified  by  subject  matter  experts.  The  analysis  of 
existing  course  documentation  is  done  to  determine  if  all  or  portions  of  the  analy*.  phiw  and 
other  phases  have  already  been  done  by  someone  else  following  the  ISO  guidelines.  As  a final 
analysis  phase  step,  the  list  of  tasks  selected  for  instruction  is  analyzed  for  the  most  suable 
instructional  setting  for  each  task. 


Beginning  with  Phase  II.  the  ISD  model  is  concerned  with  d»$igning  instruction  using  the  job 
analysis  information  from  Phase  I.  The  first  step  is  the  conversion  of  each  task  selected  for 
training  into  a terminal  learning  objective.  Each  terminal  learning  objecrive  is  then  analyzed  to 
determine  learning  objectives  and  learning  steps  necessary  for  mastery  of  the  terminal  learning 
Objective.  Tests  are  designed  to  match  the  learning  objectives.  A sample  of  students  is  tested  jp 
insure  that  their  entry  behaviors  match  the  level  of  learning  analysis.  Finally,  a sequence  of. 
instruction  is  designed  for  the  learning  objectives. 


The  instructional  development  phase  begins  with  the  classification  of  learning  objectives  by 
learning  category  so  as  to  identify  learning  guidelines  necessary  for  optimum  learning  to  take 
place.  Determining  how  instruction  is  to  be  packaged  and  presented  to  the  student  is 
accomplished  through  a media  selection  process  which  takes  into  account  such  factors  as 
learning  category  and  guideline,  media  characteristics,  training  setting  criteria,  and  costs. 
Instructional  management  plans  are  developed  to  allocate  and  manage  all  resources  for 
conducting  instruction.  Instructional  materials  are  selected  or  developed  and  tried  out.  When 
materials  have  been  validated  on  the  basis  of  empirical  data  obtained  from  groups  of  typical 
students,  the  course  is  ready  for  implementation 


Staff  training  is  required  for  the  implementation  of  the  instructional  management  plan  and  the 
instruction.  Some  key  personnel  must  be  trained  to  be  managers  in  the  specified  management 
plan.  The  instructional  staff  must  be  trained  to  conduct  the  instruction  and  collect  evaluative 
data  on  all  of  the  instructional  components.  At  the  completion  of  each  instructional  cycle, 
management  staff  should  be  able  to  use  the  collected  information  to  improve  the  instructional 
system. 


Evaluation  and  revision  of  instruction  are  carried  out  by  personnel  who  preferably  are  neither 
the  instructional  designers  nor  the  managers  of  the  course  under  study.  The  first  activity 
(internal  evaluation)  is  the  analysis  of  learner  performance  in  the  course  to  determine  instances 
of  deficient  or  irrelevant  instruction.  The  evaluation  team  then  suggests  solutions  for  the 
problems.  In  the  external  evaluation,  personnel  assess  job  task  performance  on  the  job  to 
determine  the  actual  performance  of  course  graduates  and  other  job  incumbents.  All  collected 
data,  internal  and  external,  can  be  used  as  quality  control  on  instruction  and  as  input  to  any 
phase  of  the  system  for  revision. 


THE  BLOCKS  IN  EACH  PHASE  ARE 


M 

► ANALYZE 
JOB 


EC  1 

►DEVELOP 

OBJECTIVES 


1.2 

SELECT 

TASKS/ 

FUNCTIONS 


n? 

► DEVELOP 
TESTS 


1.3 

CONSTRUCT 
► JOB 

PERFORMANCE 

MEASURES 


II.3 

.DESCRIBE 

ENTRY 

BEHAVIOR 


1.4 

ANALYZE 

'EXISTING 

COURSES 


rt.4 

^ DETERMINE 
Y SEQUENCE  & 
♦^STRUCTURE 


1.5 

SELECT 

'instructional 

SETTING 


in. 


SPECIFY 

LEARNING 

EVENTS/ 

ACTIVITIES 


ffl  .2 
SPECIFY 
INSTRUCTION 
■ MANAGEMENT 
PLAN  8i 
DELIVERY 
SYSTEM 


IH.3 

.review/select 

EXISTING 

MATERIALS 


m.4 

l m.s 

t develop 

^VALIDATE 

r INSTRUCTION 

W INSTRUCTION 

IMPLEMENT 

INSTRUCTIONAL 

MANAGEMENT 


ISC. 2 

CONDUCT 

INSTRUCTION 


CONDUCT 

INTERNAL 

EVALUATION 


12 

.CONDUCT 
' EXTERNAL 
EVALUATION 


H.3 

► REVISE 
SYSTEM 


THE  OUTCOMES  OF  THE  BLOCKS  ARE  : 


a list  of  tasks  performed  in  a particular  job. 
a list  of  tasks  selected  for  training. 

a job  performance  measure  for  each  task  selected  for  instruction, 
an  analysis  of  the  job  analysis,  task  selection,  and  performance  measure 
construction  for  any  existing  instruction  to  determine  if  these  courses  are 
usable  in  whole  or  in  part. 

selection  of  the  instructional  setting  for  task  selected  for  instruction. 


.1  . . . a learning  objective  for  and  a learning  analysis  of  each  task  selected  for 
instruction. 

.2  . . . test  items  to  measure  each  learning  objective. 

,3  . . . c test  of  entry  behaviors  to  see  if  the  original  assumptions  were  correct. 

.4.  . the  sequencing  of  all  dependent  tasks. 


. the  classification  of  learning  objectives  by  learning  category  and  the 
identification  of  appropriate  learning  guidelines. 

. the  media  selections  for  instructional  development  and  the  instructional 
management  plan  for  conducting  the  instruction. 

. the  analysis  of  pnrkages  pf  any  existing  instruction  that  meets  the  given 
learning  objectives.  , 

. the  development  of  instruction  for  all  learning  objectives  where  existing 
materials  are  riot  available. 

. field  tested  and  revised  instructional  materials. 


.1  . . documents  containing  information  on  time,  space,  student  and  instructional 
resources,  and  staff  trained  to  conduct  the  instruction. 

.2  . a completed  cycle  of  instruction  with  information  needed  to  improve  it  for 
the  succeeding  cycle. 


,1 . . . data  on  instructional  effectiveness. 

,2  . . data  on  job  performance  in  the  field. 

,3.  . . instructional  system  revised  on  basis  of  empirical  data. 


INTERSERVICE  PROCEDURES  FOR  INSTRUCTIONAL  SYSTEMS  DEVELOPMENT: 
EXECUTIVE  SUMMARY  AND  MODEL 

INTRODUCTION 

The  Interservice  Procedures  for  Instructional  Systems  Development 
(IPISD)  are  concerned  primarily  with  the  "how  to  do  it"  aspects  of 
instructional  systems  development.  In  the  overall  management  system 
context,  the  design  and  development  of  instruction  necessarily  must 
follow  an  adequate  needs  analysis.  (See  Figure  1.) 


FIGURE  1:  Relationship  of  ISD  to  Total  System 
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In  this  context,  the  term  "need"  refers  to  a measurable  discrepancy 
or  distance  between  the  actual  world  as  it  exists  now  and  the  world  as 
it  ought  to  be.  Needs  analysis  must  accurately  reflect  the  world  or 
system  as  It  now  Is  and  must  also  define  clearly  the  most  desirable 
state  of  affairs.  Discrepancies  can  be  created  through  the  discovery 
of  new  knowledge  or  the  application  of  new  technology;  e.g.,  the  avail- 
ability of  a new  weapons  system  or  through  changes  in  the  force  structure 
brought  about  oy  changes  in  strategy,  tactics,  or  changes  in  the  military 
personnel  system.  For  example,  in  the  ear  .j  1 970 1 s , the  reenlistment 
rate  of  combat  experienced  veterans  remained  at  a moderate  to  low  level. 
During  the  middle  1970' s perhaps  rs  a result  of  the  general  economic 
conditions  or  wide  scale  na.t’nn.il  altitudinal  changes,  combat  veterans 
were  reenlisting  at  a dramatically  increased  rate.  Changes  of  this 
nature  can  have  important  implications  tor  the  training  system  as  a 
whole. 

Since  the  ISO  procedures  must  follow  the  needs  analysis,  the  quality 
of  the  information  obtained  in  Phase  I of  the  ISD  procedures  will  have 
a strong  impact  on  the  success  of  the  outputs  of  the  ISD  procedures. 

If  a performance  requirement  is  overdefinea  or  underdefined  based  on 
analyses  of  field  conditions,  a new  weapons  system,  or  other  source,  the 
ISD  procedures  will  tend  to  exaggerate  the  error.  If  training  is  speci- 
fied where  no  training  is  required,  resources  will  not  be  well  managed. 

On  the  other  hand,  if  no  training  is  specified  where  training  it  an 
appropriate  solution,  the  organizational  mission  could  suffer  drastically. 

This  volume  contains  the  Executive  Summary  and  Model  of  a five-phase 
manual  wnich  presents  approved  procedures  and  techniques  for  interservice 
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instructional  systems  development.  It  provides  an  introduction  and 
overview  to  the  application  and  management  of  ISD  and  describes  the 
foundations  upon  which  ISD  is  based. 

Instructional  systems  development  is  the  broad  application  of 
the  systems  approach  to  training.  The  model  upon  which  this  manual 
is  based  has  five  phases  as  shown  in  the  fold-out  at  the  beginning  of 
this  volume.  Each  of  the  phases  is  a separate  and  distinct  function 
which  could  be  carried  out  successively  by  one  person,  or  each  of  the 
steps  could  be  assigned  to  separate  individuals. 

The  IPISD  model  is  based  on  the  following  nine  assumptions: 

1.  The  mission  of  a military  instructional  system  is  to 
determine  instructional  needs  and  priorities,  to  develop 
effective  and  efficient  solutions  to  achieving  these 
needs,  to  implement  these  solutions  in  a competent 
manner,  and  to  assess  the  degrees  to  which  the  output 

of  the  system  meets  the  specified  needs. 

2.  There  are  alternative  approaches  to  the  solution  of 
instructional  problems  which  are  differentially  re- 
sponsive to  specific  environmental  constraints  found  in 
the  Armed  Forces. 

3.  The  existing  large  body  of  research  and  development  in 
learning,  instruction,  and  management  techniques  may  pro- 
vide the  basis  for  significantly  improved  instruction. 

4.  A systems  approach  to  the  process  and  procedures  of 
instruction  is  the  most  effective  current  means  of 
evaluating,  developing,  and  implementing  these  alternatives. 


5.  Regardless  of  the  complexity  of  the  job  tasks  to  be 
performed,  the  Instructional  system  should  optimize  the 
proportion  of  entering  students  who  meet  acceptable  job 
task  performance  standards  by  the  end  of  instruction. 

6.  Planned  technical  and  management  change  In  the  operation 
of  the  Instructional  system  will  be  a continuing  require- 
ment. 

7.  Individuals  differ  in  their  abilities,  achievement, 
motivation  and  rate  of  learning  and  an  instructional 
system  must  accomodate  these  differences  to  capitalize 
on  the  opportunity  for  increasing  the  effectiveness  and 
efficiency  of  instruction. 

8.  Two  or  more  equally  successful  alternative  solutions  can 
be  found  for  any  instructional  problem,  and  these  solu- 
tions will  differ  in  cost. 

9.  Intensive  and  recurring  training  of  managers  and  instruc- 
tional developers  in  the  application  of  The  Interservice 
Procedures  for  Instructional  Systems  Development  represents 
a direct  first  step  toward  achievement  of  this  mission. 

Based  on  these  assumptions,  this  document  describes  the  functions 
necessary  to  analyze  instructional  needs;  design,  develop,  and  implement 
instruction;  and  maintain  quality  control  of  instruction.  This  document 
also  recommends  a sequential  relationship  of  these  functional  steps. 

Phase  I,  ANALYZE,  presents  procedures  for  defining  what  jobs  are, 
breaking  these  down  into  statements  of  tasks,  and  using  numerical  tech- 
niques to  combine  the  best  judgment  of  experienced  professionals  to  select 
tasks  for  training.  Phase  I also  presents  processes  for  construction  of 
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job  performance  measures  and  the  sharing  of  occupational  and  training 
information  within  and  among  the  services.  It  provides  a rationale 
for  deciding  whether  tasks  should  be  trained  in  schools,  on  the  j6b, 
or  elsewhere,  and  also  requires  consideration  of  the  interaction 
between  training  and  career  progression. 

Phase  II,  DESIGN,  of  the  manual  deals  specifically  with  the  design 
aspects  of  the  training  program  within  selected  settings.  Design  here 
is  considered  in  the  architectural  sense  in  which  the  form  and  speci- 
fications for  training  are  laid  down  in  careful  detail.  Phase  II 
reviews  the  considerations  relating  to  entry  behavior  of  two  separate 
kinds:  general  ability,  and  prior  experience.  A rationale  is  pre- 

sented for  establishing  requirements  based  on  the  realistic  eval- 
uation of  both  of  these  factors. 

Phase  III,  DEVELOPMENT,  refers  to  the  actual  preparation  of  instruc- 
tion. Determinations  are  made  about  how  the  students  shall  be  managed, 
the  kinds  of  learning  experiences  they  will  have,  the  activities  in 
which  they  will  engage,  and  the  form  and  content  of  the  instructional 
delivery  system.  Techniques  are  presented  for  the  careful  review  and 
adaptation  of  existing  materials.  Procedures  for  the  systematic  design 
of  instruction  which  can  be  delivered  in  a variety  of  media  are  also 
included.  Phase  III  terminates  with  a carefully  developed  procedure 
for  testing  and  evaluating  the  instruction  to  Insure  that  its  perfor- 
mance meets  expectations. 

Phase  IV,  IMPLEMENTATION,  specifically  treats  the  necessary  steps 
to  implement  the  instruction  according  to  the  plan  developed  in  Phase  III. 
Two  important  steps  highlight  Phase  IV,  that  of  training  the  staff  in 
the  procedures  and  problems  unique  to  the  specific  instruction  and 
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actually  bringing  the  instruction  on-line  and  operating  it.  The  Phase 
IV  effort  continues  as  long  as  there  is  a need  for  the  instruction. 

Phase  V,  CONTROL,  deals  with  procedures  and  techniques  for  main- 
taining Instructional  quality  control  standards  and  for  providing  data 
from  Internal  and  external  sources  upon  which  revision  decisions  can 

♦ 

be  based.  Data  collection,  evaluation  of  the  data,  and  decision  making 
about  the  implications  of  the  data  represent  the  three  principal 
functions  described  in  Phase  V.  Emphasis  is  placed  on  the  importance 
of  determining  whether  the  trainees  are  learning  what  was  intended,  and 
upon  determining  whether  what  they  have  learned  is  of  the  expected  ben- 
efit to  the  receiving  command.  A negative  answer  to  either  of  these 
would  suggest  revisions  in  the  content  or  procedures  in  order  to  make 
the  instruction  meet  the  need  it  is  intended  to  serve. 

How  Does  ISP  Differ  From  Existing  Practice 

One  way  to  Indicate  the  difference  between  ISO  and  existing  practices 
is  to  point  out  that  there  are  currently  a number  of  existing  practices, 
some  of  which  represent  excellent  applications  of  ISD.  There  are  outstanding 
examples  of  well-conceived  and  delivered  instruction  available  within  the 
interservice  training  community.  However,  these  efforts  do  not  represent  a 
very  large  fraction  of  the  total  interservice  training  establishment.  One  pur- 
pose of  this  manual  is  to  establish  interservice  standards  for  the  design,  develop 
ment  and  delivery  of  instruction  which  will  meet  state-of-the-art  specifications. 

An  important  difference  between  ISD  and  more  traditional  forms  of 
instruction  is  that  the  ISD  process,  through  occupational  surveys  and 
job  analyses  requires  the  thoughtful  selection  of  what  is  to  be  trained 
based  on  solid  job  data  from  the  field.  This  practice  tends  to  insure 
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that  training  will  be  provided  for  those  tasks  most  critical  to 
adequate  job  performance,  and  that  training  will  not  be  wasted  on 
tasks  which  have  a low  probability  of  meeting  Immediate' needs  or 
critical  long-term  needs. 

A second  important  difference  between  traditional  schools  and  ISD 
procedures  is  the  consideration  of  how  training  is  to  be  conducted. 

The  recent  past  has  seen  a number  of  innovations  in  approaches  to 
training  all  of  which  are  either  as  good  or  better  than  traditional 
methodology.  The  generation  and  application  of  alternative  training 
methodology  is  required  in  the  ISD  process,  it  is  not  assumed  that  all 
training  will  be  platform  instruction. 

A third  critical  difference  between  traditional  practice  and  ISD  is 
the  ISD  use  of  test  data  based  on  absolute  standards  of  performance  and 
the  use  of  that  data  to  grade  students  and  to  judge  the  quality  of  the 
instruction.  Tnere  are  specific  objectives  that  courses  are  planned  to 
meet,  and  ISD  requires  that  courses  be  evaluated  on  their  ability  to 
meet  those  stated  objectives,  and  be  revised;  if  they  fail  to  do  so. 
Finally,  the  ISD  process  requires  the  application  of  modern  tech- 

i 

nology  to  the  fullest  degree  possible  iri  order  to  optimize  training 

i 

effectiveness,  efficiency  and  cost.  Consideration  is  given  to  the  rel- 
ative value  of  training  compared  to  its  cos|t,  and  whether  the  output  of 

i 

the  training  system  is  worth  the'  investment)  of  time  and  resources  required 
to  produce  that  output.  A unique  feature  wjhich  distinguishes  ISD  from 
more  traditional  approaches  is  that  course  time  or  cost  reductions  are 

brought  about  not  by  the  elimination  of  content  or  the  reduction  of 

I 

service  but  through  the  application  of  a technology  to  achieve  expected 
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performance  with  fewer  resources.  The  application  of  unit  cost  and 
unit  time  reduction  techniques  often  have  produced  dramatic  results. 

Bases  for  ISP 

Instructional  systems  development  has  grown  out  of  basic  research 
in  three  separate  areas:  management  sciences,  communication  sciences, 
and  behavioral  sciences.  Examples  of  basic  research  areas  in  the 
management  sciences  include:  job  analysis,  occupational  survey 

techniques,  decision  theory,  cost  effectiveness  models,  and  computer 
technology. 

From  the  communications  sciences,  research  in  communications, 
electronics,  and  media  utilization  have  produced  a wide  variety  of 
alternative  techniques  and  procedures  for  accomplishing  instructional 
objectives. 

There  are  three  important  areas  of  research  in  the  behavioral  sci- 
ences which  have  yielded  results  that  are  useful  in  ISO.  Learning 
research  has  provided  a solid  foundation  for  the  design  of  alternative 
approaches  to  instruction.  Measurement  and  evaluation  of  behavior  has 
matured  to  the  point  that  it  is  possible  to  have  great  confidence  in  the 
measurement  and  evaluation  procedures.  And,  the  recent  past  has  seen 
a large  variety  of  instructional  design  and  management  approaches  which 
have  yielded  impressive  results. 

These  contributions  from  the  management,  communications,  and  be- 
havioral sciences  allow  for  the  development  of  ISO  technology.  The  ISD 
process  Includes  the  capability  for  specific  research  and  development 
to  resolve  existing  problems.  In  addition,  because  it  provides  for  so 
many  alternatives  to  traditional  forms  of  instruction,  the  ISD  process 
allows  for  the  analysis  and  use  of  existing  research  bases. 
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Potential  Benefits  of  ISP 

Based  on  a large  number  of  successful  demonstrations,  there  Is 
now  empirical  evidence  that  competent  use  of  the  ISD  approach  can 
greatly  improve  training  in  at  least  three  distinct  ways: 

1.  Effectiveness.  Through  the  design  and  development 
procedures,  a careful  selection  of  what  is  to  be  trained, 
the  measurement  and  evaluation  of  training,  and  the 
revision  of  the  training  program  until  it  meets  its  ob- 
jectives should  greatly  increase  training  effectiveness. 

2.  Efficiency.  Several  military  applications  of  ISD  have 
indicated  that  effective  instruction  can  be  offered  in  a 
much  more  time-efficient  way  than  has  been  true  in  the 
past.  The  application  of  ISD  procedures  to  instruction  in 
order  to  make  it  more  time-efficient  has  paid  oft  hand- 
somely. 

3.  Costs.  It  is  not  reasonable  to  believe  that  the  use  of 
ISD  procedures  will  always  result  in  lower  costs.  It 
is  unrealistic  to  expect  lower  costs  per  student  on  all 
existing  completely  effective  courses.  However,  the  ISD 
procedure  does  provide  a systematic  way  of  viewing  costs  of 
training  and  considering  whether  additional  resources  are 
justified  in  view  of  the  output. 

There  have  been  many  demonstrations  that  combinations  of  effectiveness, 
time  - efficiency,  and  cost  considerations  have  yielded  impressive  results, 
particularly  when  they  have  been  considered  in  the  context  of  making 
alternative  investment  decisions.  Investments  in  technology  for  certain 
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long  high-flow  courses  have  demonstrated  Improvements  In  cost  per  stu- 
dent, time  required  to  complete,  and  increased  effectiveness.  These 
results  have  been  obtained  on  large  systems  which  use  advanced  simu- 
lators and  also  In  areas  of  training  which  use  no  hardware  at  all. 

The  common  element  is  the  procedure  and  approach,  not  the  hardware  or 
equipment. 

A vitally  important  management  function  is  the  accurate  collection 
and  use  of  cost  data.  Such  data  permit  cost  comparisons  as  soon  as 
they  are  available,  and,  more  importantly,  make  the  conduct  of  cost- 
effectiveness  studies  more  likely  and  their  conclusions  more  accurate. 

One  cost  model,  produced  by  the  Navy's  Training  Analysis  Evaluation 
Group,  called  TECEP  (Training  Effectiveness  and  Cost  Effectiveness 
Prediction)  is  suggested  as  a companion  volume  to  the  interservice  man- 
uals. The  TECEP  model,  or  an  adaptation  or  improvement  of  it,  can  be 
used  by  local  commands  to  accumulate  and  use  cost  data. 


MAJOR  COMPONENTS  OF  THE  IPISD  PACKAGE 


The  Interservice  Procedures  for  Instructional  Systems  Development 
Package  contains  components  essential  to  produce  the  coordinated 
effort  between  senior  management,  middle  management,  and  individuals 
who  work  in  ISD.  The  complete  training  package  contains  the  following 
elements: 

1 . An  Instructional  Systems  Development  Executive  Summary 
and  Model  with  explanation  and  rationale  is  the  intro- 
duction to  the  manuals. 

2.  A multi  volume  manual  covers  the  procedures  for  instruc- 
tional systems  analysis,  design,  development,  implemen- 
tation and  control. 

3.  An  adjunct  instructional  workbook  which  is  to  be  used  in 
conjunction  with  the  multi  volume  manual.  The  workbook 
contains  practice  exercises,  forms,  and  procedures  to  be 
followed;  methodology  to  be  employed;  and  the  necessary 
evaluation  items  to  enable  the  student  to  do  independent 
study. 

4.  Mediated  workshop  materials  to  be  used  either  with  indi- 
viduals or  groups.  These  will  supplement  the  instructional 
program  by  teaching  trainees  the  necessary  knowledges  and 
skills  described  in  the  model.  These  mediated  workshop 
materials  consist  of  slide/tape  presentations  combined 
with  exercises  from  the  adjunct  workbook.  Workshop  in- 
structors will  provide  regular  feedback  to  the  trainee 

on  his  progress  in  achieving  the  objectives. 
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4 workshop  directors  handbook  describes  the  procedural 
steps  necessary  to  conduct  short  or  longer  term  work- 
shops utilizing  elements  1,  2,  3,  and  4.  It  contains  sug- 
gested exercises,  suggested  methods  for  adapting  the  gen- 
eral procedures  described  in  1,  2,  3,  and  4 to  local 
conditions  and  situations,  and  contains  additional  evalua- 
tion instruments  necessary  for  quality  control. 

The  package  is  a combination  of  individual  study,  group  study,  and 
review  materials,  and  is  designed  to  accomodate  a variety  of  learners 
and  institutional  requirements.  The  recommended  situation,  however, 
is  for  an  individual  first  to  attend  a directed  workshop  with  a group 
of  other  trainees.  There  he  would  receive  practice  exercises,  orien- 
tation sessions,  and  informational  presentations  to  enable  him  to  grasp 
the  general  concepts  of  ISO.  Further,  he  would  be  taught  how  to  access 
and  use  the  information  contained  in  the  multivolume  manual,  how  to 
use  existing  referenced  Armed  Forces  publications,  and  how  to  continue 
to  improve  his  skills  at  the  conclusion  of  the  workshop. 

Both  the  multivolume  manual  and  the  adjunct  workbook  contain  ref- 
erences to  significant  existing  publications  dealing  with  the  specific 
topics  covered.  Each  of  the  Armed  Forces  has  published  manuals  on 
procedures  for  the  development  of  training,  and  each  step  in  the  Model 
utilizes  references  to  specific  publications  available  to  the  Armed 
Forces  schools. 

The  multivolume  manual  serves  the  function  of  information 
storage;  that  is,  as  a set  of  reference  works.  The  manual  is  designed 
in  this  fashion  to  enable  Individual  commands  to  add  specific  books 
and  manuals  to  its  basic  list  of  references  in  order  to  make  the  package 
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adaptable  to  local  needs.  The  adjunct  workbook  is  loose-leaf  in 
format,  individual  commands  are  urged  to  add  or  subtract  materials, 
problems,  and  examples  to  meet  local  needs.  The  combination  of 
the  workbook  and  multivolume  manual  will  he  most  useful  to  those 
individuals  who  have  independent  study  skills. 

The  workshop  slide/tape  is  organized  to  provide  a guided  overview 
for  a workshop  briefing  or  individual  learning  center  setting.  The 
package  can  also  be  used  in  combination  with  instructors  for  larger 
numbers  of  individuals.  The  training  workshops  teach  the  use  of  the 
multi  volume  manual  and  rely  on  the  adjunct  workbook  for  the  provision 
of  practice  exercises  and  self-evaluation. 

These  workshops  provide  either  intensive  training  in  specific 
areas  contained  in  the  ISD  manuals,  or  a complete  overview  of  the  Model 
and  its  elements  for  those  individuals  having  need  for  a more  general 
level  of  knowledge. 

The  last  element  in  the  package  is  a workshop  directors  handbook 
which  contains  administrative  instructions  for  each  of  the  components 
of  the  IPISD  package.  It  also  contains  practice  exercises  and  suggested 
instructional  development  problems  for  which  alternative  solutions  are 
available  and  which  can  be  designed  as  case  studies  in  a workshop 
setting. 

The  materials  are  usable  in  group  mode  workshops  of  two  or  three 
weeks  duration  for  individuals  who  need  to  learn  all  of  the  ISD  skills. 
The  managers  workshop  is  of  approximately  one  week's  duration  and 
contains  the  major  concepts,  information  sources,  and  strategies  for 
selecting  alternative  forms  of  instruction. 
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The  Senior  Managers  Workshop  lasts  approximately  two  days  and  deals 
with  the  longer  range  management,  planning,  budgeting,  and  strategy 
considerations  necessary  for  the  continuation  of  the  new  training 
system. 

A design  consideration  permeating  the  IPISD  Package  is  that  all 
Individuals  within  a command  who  are  concerned  with  any  of  the  aspects 
of  Instructional  development  should  receive  training  at  or  near  the 
same  time  in  order  to  establish  a common  basis  for  communicating 
about  methods,  procedures,  and  evi.  „*.tive  techniques  that  are  to  be 
used. 

It  is  anticipated  that  the  IPISD  Package  will  continue  to  be 
revised  in  the  future  both  as  an  interservice  training  approach  and 
within  each  of  the  services  to  meet  the  requirements  of  local  situa- 
tions. As  experience  is  gained  with  the  use  of  the  IPISD  Program 
it  is  anticipated  that  subsequent  editions  will  continue  to  improve. 


BLOCK  I. 1:  ANALY7E  JOB 


Introduction 

Job  analysis  is  a highly  systematic  procedure  for  finding  out 
exactly  what  people  do  when  they  do  their  job,  the  order  in  which  they 
do  it,  the  conditions  under  which  they  must  do  it,  and  the  level  of 
skill  or  performance  deemed  adequate  in  the  job.  Job  analyses  are 
oridnarily  conducted  by  observing  a job  incumbent  do  his  job  and 
making  detailed  notes  on  these  observations,  by  interviewing  job  in- 
cumbents about  what  they  do  on  their  job,  and  by  occupational  survey 
methods . 

Job  analysis  requirements  are  most  often  generated  by  the  install- 
ation of  new  weapons  or  procedural  systems,  or  to  update  knowledge  about 
existing  systems.  The  extent  that  one  used  the  interview  method,  the 
observation  method,  or  the  occupational  survey  method  depends  on  the 
nature  of  the  job  being  analysed,  the  job  data  already  available,  and 
the  availability  of  analysis  resources. 

Rationale 

Job  analyses  have  multipurpose  uses  in  training,  in  manning  require- 
ments, and  for  personnel  selection.  Effective  occupational  survey 
techniques,  combined  with  well  constructed  questionnaires,  can  yield  an 
immense  amount  of  valuable  data  regarding  the  task  difficulty,  rank  and 
experience  level  of  performers,  and  the  supervisory  requirements  for 
jobs. 

In  ISO,  the  principal  concern  is  with  training.  Specifically,  one 
of  the  more  important  purposes  of  job  analysis  is  to  concentrate  resources 
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on  the  high  priority  jobs  and  the  tasks  which  make  up  those  jobs,  and 
to  reduce  resources  which  may  presently  be  concentrat  d on  tasks  of 
lower  priority.  In  order  to  know  how  to  train  individuals  to  do  jobs, 
it  is  necessary  to  have  highly  detailed  information  about  how  the  jobs 
are  done.  The  training  system  is  intended  to  be  totally  supportive  of 
the  job  world  and  as  such  must  be  based  on  highly  detailed  infor- 
mation. 

In  ISO,  job  analysis  effort  probably  represents  the  greatest  invest- 
ment of  time  and  money  of  any  of  the  initial  steps  of  course  design. 

This  investment,  when  it  is  properly  managed,  yields  extremely  impres- 
sive payoffs  in  training  effectiveness  and  cost  efficiency.  These  pay- 
offs are  principally  due  to  the  organization  of  training  and  aimed  at 
concentrating  on  the  important  aspects  of  the  job  and  selectively 
ignoring  the  unimportant  parts  of  the  job. 

Inputs 

The  inputs  to  the  Analyze  Job  block  ordinarily  come  from: 

1.  needs  analyses  conducted  by  higher  headquarters, 

2.  the  installation  of  new  systems,  weapons,  equipment, 
or  procedures,  and 

3.  the  systematic  periodic  updating  of  Defense  Occupation 
Specialties  (DOS). 

Items  1 and  2 may  indicate  that  a job  is  being  analyzed  for  the 
first  time  in  which  case  there  will  only  be  a very  small  information 
base  for  inputs.  Item  3,  on  the  other  hand,  represents  a revision  in 
the  job  required  in  some  DOS  and  will  have  a significant  literature 
which  needs  to  be  examined.  In  addition  to  the  above  inputs,  after  a 
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job  has  been  analyzed,  and  training  has  been  designed  according  to  an 
ISO  process,  there  will  be  internal  systems  inputs  to  the  Analyze  Job 
block  provided  in  the  form  of  feedback  from  Phase  V. 

Procedures 

The  basic  procedures  employed  in  job  analyses  include  the 
fol lowing: 

1.  the  initial  development  of  a tentative  task  list.  Such 
lists  are  ordinarily  developed  by  studying  existing 
written  doctrine,  by  convening  a jury  of  experts  who  can 
detail  the  requirements  of  a specific  job,  or,  in  the  case 
of  new  jobs,  by  analyzing  similar  jobs  and  by  engaging  in 

a concerted  work  effort  with  the  developers  of  new  hardware 
and  systems. 

Once  a tentative  task  list  has  been  generated,  it  is  nec- 
essary to  decide  whether  the  observation  method  or  interview 
method  or  a combination  of  both  will  be  used  to  perform  a 
job  analyses.  Or,  in  the  case  where  it  is  possible  to  per- 
form an  adequate  occupational  survey,  the  decision  may 
be  made  to  go  directly  to  those  procedures. 

2.  The  second  principal  step  in  analyzing  jobs  is  to  authen- 
ticate the  task  list  by  going  to  actual  job  incumbents, 
either  in  person  or  by  questionnaire,  and  having  them 
verify  the  performance  of  the  tasks  on  the  lists,  and  de- 
pending on  che  manacement  decision,  provide  descriptive 
characteristics  of  those  tasks. 
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3.  Once  a task  list  has  been  developed,  It  must  be  validated. 
Validation  is  a process  through  which  a task  list  generated 
from  one  set  of  incumbents  and  supervisors  is  submitted  to 
other  sucii  people  for  verification.  This  process  Insures 
that  the  tasks  on  the  lists  are  those  actually  performed  by 
members  of  the  DOS.  Generally,  at  the  same  time--with  the 
same  questionnaire  survey--that  the  task  list  is  being  val- 
idated, data  will  be  collected  upon  which  to  base  the 
selection  decisions  to  be  made  in  the  next  block.  Details 
of  this  additional  data  to  be  collected  will  be  given  in 
the  next  block. 

4.  A final  part  of  the  process,  depending  upon  the  management 
decisions,  will  be  to  collect  the  conditions  under  which 
the  tasks  are  performed,  cues  that  initiate  and  guide  per- 
formance of  the  task,  and  the  standards  which  represent 
adequate  task  performance.  Further,  if  such  information 
is  not  already  available,  it  will  be  necessary  to  collect 
the  description  of  the  task  elements,  those  subparts  of  the 
task  necessary  to  its  performance  but  which  are  not  per- 
formed as  an  end  in  themselves. 

Outputs 

The  outputs  of  Block  1.1  will  include,  depending  upon  the  methodology 
selected,  a validated  list  of  tasks,  the  conditions  under  which  the 
tasks  are  performed,  the  initiating  cues,  the  standards  to  which  tasks 
must  be  performed,  and  documentation  of  the  task  elements. 


In  addition,  a documentation  report  will  be  prepared  which  de- 
scribes the  methodology  employed  in  performing  the  job  analysis,  the 
principal  findings,  and  the  principal  difficulties  encountered. 


Management  Decisions 

- The  important  management  decisions  in  analyzing  jobs  will  be  in 
the  following  areas: 

1.  Approval  of  plan.  The  first  action  in  the  conduct  of  a 
job  analysis  is  to  develop  a plan  which  describes  the 
method  of  task  list  development,  the  procedures  for  val- 
idation of  the  task  list,  and  the  procedures  for  obtaining 
detailed  information  on  how  the  tasks  are  performed. 

2.  Support  of  plan.  Since  a large  number  of  job  incumbents, 
their  supervisors,  and  others  are  usually  involved  in 
job  analysis,  management  support  must  be  provided  in 

a number  of  locations  to  assure  that  the  involved  in- 
dividuals can  make  maximum  contributions  to  the  job 
analysis  effort  with  minimum  interference  with  their 
regular  duties. 

3.  Monitoring  of  plan.  Since  no  plan  can  make  provisions  for 
every  possible  occurance,  revisions  may  have  to  be  made  and 
approved.  Management  must  assure  that  any  deviations  from 
plan  are  justified  and  appropriate. 

Because  each  of  these  decisions  requires  the  commitment  of  resources,, 
making  the  decisions  will  require  discussions  of  alternatives  and  the 
trade-offs  among  what  is  desired,  what  is  available,  and  what  can  be 
afforded.  As  the  basis  for  all  that  follows,  it  is  particularly  essential 
that  the  job  analysis  output  be  accurate  and  complete. 
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BLOCK  1.2:  SELECT  TASKS/ FUNCTION'S 

Introduction 

The  second  step  in  the  ISO  procedure  is  the  selection  of  tasks 
for  which  training  will  be  given.  Information  collected  during  the 
job  analysis  and  occupational  survey  procedures  in  the  prior  block  is. 
used  to  decide  which  tasks  are  of  sufficient  importance  to  train. 

Selecting  Tasks/Functions  represents  a critical  step  in. the  ISD 
process  because  it  is  at  this  point  that  those  tasks  selected  will 
obligate  resources  throughout  the  entire  ISD  procedure.  Any  task 
rejected  is,  for  the  training  establishment,  no  longer  a factor. 

Rationale 

Fundamental  to  the  notion  of  selecting  tasks  for  training  is  the 
assumption  that  there  rarely  will  be  enough  time  or  resources  to  train 
everything  that  might  be  desirable  to  train.  While  it  may  be  possible 
under  some  highly  threatening  national  circumstances  to  obtain  financial 
resources,  it  is  likely  that  even  then  the  available  tW  and  personnel 
resources  cannot  be  directed  to  the  training  effort.  In  any  event  it 
Is  assumed  that  because  of  these  resource  constraints,  decisions  will 
have  to  be  made  in  terms  of  the  priorities  assigned  to  the  various  tasks. 

Even  if  there  were  unlimited  time  and  resources,  some  tasks  would 
have  to  be  trained  before  others  because  of  the  serial  nature  of  work. 

One  way  of  conceptualizing  this  part  of  the  process  is  to  list  the  tasks 
in  the  order  that  they  will  be  trained. 

The  past  few  years  have  seen  improvements  in  the  techniques  and  pro- 
cedures available  for  task  selection.  Exceptionally  good  data  is  now 
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available  upon  which  to  base  decisions,  and  it  is  no  longer  necessary 

or  desirable  to  rely  on  guesses  about  which  tasks  most  need  training 

In  order  for  the  job  to  be  done  adequately.  Oust  as  a job  analysis 

describes  what  It  is  that  Individuals  do  when  they  do  their  job.  In 

sufficient  detail  to  allow  for  good  decisions  to  be  made  about 
curriculum,  selecting  tasks/functions  allows  clear  decisions  to  be 
made  about  the  relative  Importance  of  the  various  tasks  compared  to 
the  resources  available. 

Inputs 

The  output  from  Block  1.1,  Analyze  Job,  is  the  fundamental  input 
to  Block  1.2.  This  output  will  c.ome  in  the  form  of  a documentation 
report  which  will  Include  a task  list,  and  possibly  the  conditions 
initiating  cues,  standards,  and  task  elements.  However,  depending  on 
management  decisions,  only  the  unvalidated  task  list  is  essential  to 
beginning  work  in  this  block.  This  is  because  the  validation  procedure 
in  Block  1.1  generally  is  done  together  with  the  collection  of  data 
for  making  selection  decisions  in  this  block. 

Processes 

The  first  important  process  in  selecting  tasks  is  the  identification 
of  the  criteria  that  will  be  used  for  selection.  Task  criteria  which 
have  been  used  in  the  past  include  the  following: 

1.  The  percentage  performing.  This  refers  to  the  percentage 
of  people  who  completed  the  questionnaire  or  interview 
who  said  they  actually  performed  the  task. 

2.  Percent  time  spent  performing.  Of  those  who  performed  the 
task,  this  is  the  percent  of  their  total  work  time  they 

spend  on  the  task. 

\ .... 
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3.  Probable  consequence  of  Inadequate  performance.  Many 
tasks  are  of  such  a nature  that  if  they  are  performed  1m- 
properly  or  inadequately  the  consequences  to  mission 
success  accomplishment,  equipment,  or  personnel  are  ex- 
tremely severe. 

4.  Task  delay  tolerance.  Each  task  has  an  initiating  cue. 

This  refers  to  the  time  when  it  becomes  appropriate  or 
necessary  to  perforin  the  task.  Task  delay  tolerance 
refers  to  the  amount  of  time  that  can  be  tolerated 
between  the  initiating  cue  and  the  actual  performance. 

5.  Frequency  of  performance.  This  refers  to  the  likelihood 
that  a task  will  be  performed  frequently  enough  to  require 
training. 

6.  Task  learning  difficulty,  learning  difficulty  refers  to 
the  amount  of  time  required  to  learn  the  task  to  an  accept- 
able level  of  performance. 

7.  Probability  of  deficient  performance.  This  refers  to 
the  likelihood  that  job  incumbents  will  fail  to  perform 
a task  adequately. 

8.  Immediacy  of  performance.  This  refers  to  the  time  lapse 
between  an  individual's  assignment  to  a job  and  the  time 
when  he  is  expected  to  perform  the  task. 

There  are  many  other  criteria  which  have  been  used  by  different 
people.  However,  it  is  important  that  criteria  be  selected  that  reflect 
the  needs  of  the  using  command.  Then  when  data  are  collected  on  how  each 
task  is  rated  on  these  criteria,  the  best  professional  military  judgment 
will  have  been  obtained  upon  which  to  base  selection  of  tasks  for  training. 
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The  next  part  of  the  process  Is  to  organize  tasks,  based  on  the 
criteria  determined  earlier  Into  those  for  which  training  is  not  re- 
quired, those  for  which  training  must  be  provided,  and  those  where 
training  is  optional  depending  on  availability  of  training  resources. 

Finally,  based  on  the  available  resources,  selection  criteria,  and 
the  composition  of  the  task  list,  recoimtendations  are  made  on  which 
tasks  should  be  trained  and  which  tasks  should  be  learned  by  individual 
on  their  own. 

Outputs 

The  primary  output  will  be  a list  of  tasks  selected  for  training. 
There  must  be  considerable  interaction  between  training  resource  mana- 
gers and  task  selection  managers  to  prevent  select  n of  more  tasks 
for  training  than  resources  will  permit.  It  is  also  possible  to  have 
a greater  capability  for  training  than  task  selection  criteria  would 
dictate.  Either  of  these  situations  should  call  for  a reallocation  of 
resources  to  more  important  programs. 

A second  output  is  the  complete  documentation  of  the  methods  and 
procedures  followed  and  the  assumptions  and  considerations  which  govern 
the  output.  This  part  of  the  documentation  will  be  extremely  valuable 
the  next  time  job  analysis  is  performed  in  the  DOS. 

Management  Decisions 

The  important  management  decisions  required  in  selecting  tasks/ 
functions  are: 

1.  Provision  of  inputs  to,  and  approval  of  the  task  criteria 
to  be  used  in  selecting  tasks  for  training. 


2.  Determination  of  resource  allocations  for  development  and 
Implementation  of  training  programs. 

3.  Approval  of  tasks  selected  for  training.  These  are  critical 
management  decisions  since  they  obligate  resources  to  train- 
ing the  chosen  tasks. 


BLOCK  1.3:  CONSTRUCT  PERFORMANCE  MEASURES 
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Introduction 

A third  critical  function  In  the  ISD  process  is  the  construction 
of  Job  Performance  Measures  (JPMs)  for  each  of  the  tasks  selected  for 
training.  Job  Performance  Measures  may  be  combined  to  make  up  the  tests 
which  tell  whether  or  how  well  a person  can  perform  a job. 

Each  task  selected  for  training  must  have  an  associated  JPM  which 
is  the  best  available  indicator  of  performance.  Job  Performance  Mea- 
sures are  used  to  determine  job  proficiency,  design  and  evaluate  training, 
and  maintain  quality  control.  They  are  used  by  unit  commanders,  the 
personnel  system,  and  by  the  training  system  when  there  is  a need  to 
measure  job  performance  or  to  design  or  evaluate  training. 

Rationale 

Recent  developments,  arising  out  of  the  management  of  the  military 
system,  and  decisions  by  the  United  States  Supreme  Court  have  made  it 
mandatory  that  Job  Performance  Measures  of  the  highest  possible  technical 
quality  be  developed  for  each  job  in  the  service.  In  addition  to  the 
obvious  direct  benefits  to  military  training  and  operations,  good  JPMs 
can  eliminate  unnecessarily  restrictive  standards  in  many  DOSs  and,  at 
the  same  time,  eliminate  charges  of  real  or  apparent  discrimination  in 
job  assignments. 

The  development  of  JPMs  is  a highly  technical  professional  area  in 
which  one  finds  many  measurement  specialists.  From  time  to  time  it  will 
be  necessary  to  obtain  inputs  from  these  specialists  in  order  to  insure 
the  quality  and  utility  of  the  JPMs. 
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In  job  analysis,  it  is  necessary  to  document  the  conditions  under 
which  tasks  are  performed  and  to  obtain  real  or  estimated  standards 
which  must  be  met  in  order  to  perform  the  tasks  satisfactorily. 

Here,  a determination  is  made  to  see  how  best  to  measure  the  docu- 
mented requirements  for  any  given  task. 

It  is  in  this  block  that  a clear  statement  is  made  about  what  is 
expected  In  job  performance.  Systems  respond  to  those  elements  within 
them  which  are  measured  and  used  as  the  basis  for  evaluation.  What  is 
measured  by  the  JPM  will  be  that  which  is  aspired  to  by  the  incumbents. 

The  development. of  JPMs  is  a difficult  technical  assignment 
primarily  because  of  problems  with: 

1.  Validity.  Validity  is  the  degree  to  which  a JPM  measures 
what  it  is  Intended  to  measure.  Often,  particularly 

with  combat  tasks,  validity  is  difficult  if  not  impossible 
to  ascertain. 

2.  Fidelity.  The  administration  of  performance  tests  in 
many  DOSs  is  extremely  difficult.  It  is  not  always 
possible,  at  any  reasonable  price,  to  maintain  high 
fidelity. 

3.  Administration.  The  administration  of  JPMs  in  some  DOSs 
can  present  complex  problems  of  logistics.  In  some  in- 
stances there  will  be  test  problems  involving  the  use  of 
heavy,  complicated,  or  delicate  equipment  which  may  or 
may  not  be  continuously  available.  Many  people  may  be 
needed . 

4.  Costs.  Obtaining  high  validity  and  fidelity  may  be  more 
expensive  than  the  probable  benefits. 
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5.  Time.  It  may  take  more  time  to  administer  some  tests 
than  is  practical  under  normal  circumstances. 

While  it  is  absolutely  essential  that  adequate  resources  be  de- 
voted to  the  development  of  JPMs,  it  also  must  be  realized  that  there 
are  limitations  and  constraints  which  necessarily  reduce  their  total 
usefulness.  The  degree  to  which  these  constraints  and  restrictions 
can  be  overcome  will  determine  in  part  the  relative  success  of  the 
JPMs  through  time. 

Since  JPMs  operationally  define  the  task  in  question,  their  de- 
velopment must  maintain  as  high  a degree  of  validity  and  fidelity  as  is 
practical.  In  this  manual,  an  operational  definition  refers  to  the 
specific  observable  actions  or  symbols  which,  when  they  occur,  are 
said  to  be  evidence  that  the  phenomenon  exists.  For  example,  if  the 
task  requires  fixing  a component  and  the  JPM  requires  describing  how  it 
works,  this  change  defines  the  task  in  terms  of  what  is  measured,  not 
necessarily  in  terms  of  what  is  desired.  Virtually  any  reduction  in 
the  quality  of  a JPM  will  mean  the  overall  performance  status  of  the 
entire  service  is  reduced  by  that  amount. 

Inputs 

The  inputs  to  this  block  will  be  a list  of  tasks  that  have  been 
selected  for  training,  along  with  the  conditions,  cues,  standards,  and 
elements . 

Processes 

There  are  a variety  of  processes  which  must  be  performed  in  order  to 
insure  that  the  output  of  Block  1.3  is  adequate.  The  output  must  be 
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j.  ‘h  \ adequate  not  only  in  terms  of  the  JPMs  that  are  produced  but  also  in 

\ terms  of  the  considerations  and  recommendations  for  simulators. 

f.'..  Many  systems,  particularly  those  procured  outside  the  service 

• ■ from  contractors  and  suppliers,  will  have  a simulator  analysis  prior 

’j' 

■i  ■ to  the  time  the  development  contract  is  awarded.  On  the  other  hand,  the 

i ' , ; need  for  simulators  not  contracted  as  a part  of  the  hardware  system 

i 

must  be  determined  through  careful  analysis  of  existing  operations.  The 
l '*  method  used  to  determine  a need  for  simulators,  would  be  that  of  com- 

' paring  the  possible  performance  on  the  available  JPMs  with  that  level 

of  performance  required  on  the  actual  job.  If  realistic  field  conditions 
! can  be  created  and  adequate  feedback  given  without  the  use  of  simulators, 

! I 

■ then  they  may  not  be  necessary.  If  the  job  cal,ls  for  a level  of  practice 

I 

and  precision  far  beyond  that  which  it  is  now  possible  to  provide,  it 
may  be  necessary  to  develop  a simulation  device. 

> | 

A second  requirement  in  the  development  of  JPMs  is  to  determine  the 
\ level  of  fidelity  that  is  going  to  be  built  into  the  JPM.  A high  fidelity 

; JPM  is  one  which  closely  resembles  the  actions,  conditions,  cues,  and 

standards  actually  performed  on  the  job. 

Another  process  in  JPM  development  is  the  analysis  of  time  and 
cost  required  to  administer  the  JPM  to  members  of  the  DOS.  Excessively 

c | high  costs  and  long  periods  of  time  for  test  administration  must  be 

Sr  I 

, | avoided. 

r j 

% j The  JPM  developer  must  decide  whether  he  is  going  to  use  a product, 

*'•  ! process,  or  a combination  method  of  evaluation.  The  selection  is  governed 

| by  the  characteristics  of  the  task  and  cost  and  time  considerations, 

j The  JPM  must  be  as  objective  as  possible  and  yield  hard  performance 

j ■ data  which  can  readily  be  interpreted. 
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Once  these  decisions  and  determinations  have  been  made,  it  is  pos- 
sible to  begin  preparation  of  the  OPMs.  Each  JPM  is  associated  with 
one  task.  It  measures  enough  parts  of  the  task  to  make  a sound 
generalization  about  task  oerformance.  In  some  Instances,  the  whole 
task  must  be  measured  to  make  a judgment  while  in  other  instances  parts 
of  the  task  will  reveal  whether  or  not  the  incumbent  can  perform. 

The  final  step  iri  development  of  JPMs  is  the  validation  of  the  JPMs 
under  field  conditions.  To  be  validated,  the  JPMs  must  be  administered 
to  job  Incumbents  and  the  JPMs  must  separate  them  into  categories: 
those  who  can  perform  the  task,  and  those  who  cannot.  If  the  JPM  does 
not  yield  an  adequate  measure  of  the  task  to  permit  accurate  decisions 
about  those  individuals  who  can  perform  and  those  who  cannot,  it 
must  be  revised  until  It  meets  those  requirements. 

Outputs 

The  outputs  of  this  block  include  a validated  Job  Performance  Measure 

for  each  task  selected  for  training.  This  JPM  then  becomes  the  official 

performance  measure  for  that  task,  and  its  inclusion  in  various  kinds 

of  tests  indicate  that  it  is  the  approved  measure  of  performance  for 

\ 

that  task  in  the  service. 

A second  output  is  the  scoring  key,  directions,  and  procedures  pack- 
age which  accompanies  the  JPM  and  provides  the  exact  steps,  procedures, 
and  resources  necessary  to  administer  the  test  appropriately. 

Management  Decisions 

Management  decision-making  requirements  in  31ock  1.3  are  quite  heavy, 
not  only  In  terms  of  the  number  of  alternatives  that  must  be  explored 
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and  chosen,  but  also  In  terms  of  the  Impact  these  decisions  have  on 
future  training  and  personnel  decisions. 

The  first  decision  that  will  need  to  be  made  by  the  manager  is 
which  of  the  JPMs  shall  be  developed  first.  Since  there  will  be  a 
rather  lengthy  list  of  tasks  for  which  JPMs  will  be  required,  these 
OPMs  must  be  ordered  and  some  done  before  others  in  order  to  accomplish 
the  work  in  a reasonable  period  of  time.  In  the  process  of  analyzing 
the  tasks  for  performance,  it  will  become  apparent  that  some  tasks  will 
be  relatively  straightforward  and  easy  to  measure  while  others  will  be 
rather  complicated  and  require  more  time.  Some  JPMs  may  require  special 
kinds  of  simulators  and  training  devices  which  will  require  a long 
lead  time  for  their  acquisition. 

A second  kind  of  decision  will  involve  a large  variety  of  resource 
and  technical  trade-offs.  The  technical  characteristics  of  the  JPM  may 
require  that  it  be  administered  under  field  conditions  in  as  realistic 
an  environment  as  possible.  On  the  other  hand,  the  time  and  resource 
constraints  may  limit  the  number  of  people  who  could  be  tested  under 
those  conditions.  It  will  be  the  manager's  responsibility  to  examine 
the  total  number  of  JPMs  required,  the  total  time  and  resource  require- 
ments, and  to  make  reasonable  trade-offs  among  these  requirements  and 
constraints . 

A third  kind  of  analysis  and  decision  will  involve  those  JPMs  in 
which  the  testing  errors  from  the  past  have  been  known  to  misclassify 
those  who  took  the  test.  That  is,  the  test  has  either  indicated  people 
were  competent  or  qualified  who  were  not,  or,  it  has  indicated  that 
qualified  people  were  not  qualified.  Either  kind  of  error  is  extremely 
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difficult  to  tolerate,  but  the  manager  must  choose  among  the  alternatives, 
decide  which  risks  he  is  willing  to  assume  and  which  he  is  not,  and 
allocate  his  resources  accordingly. 

The  last  set  of  management  decisions  involves  the  validation  process. 
Test  validation,  like  markmans'nip,  can  always  get  better.  That  is,  a 
test  can  be  revised  and  improved  until  its  validity  reaches  the  technical 
limit.  The  manager  must  decide  the  level  of  validity  that  he  will  be 
willing  to  accept  and  the  amount  of  his  resources  that  he  is  willing  to 
invest  in  achieving  that  level  of  validity.  In  those  DOSs  with  mini- 
mum risks,  one  can  be  satisfied  with  lesser  validity  than  in  those  where 
inadequate  performance  can  have  disastrous  results. 

While  this  list  does  not  exhaust  the  management  decisions  required 
in  the  development  of  OPMs,  it  does  highlight  the  significant  decisions 
that  he  will  be  forced  to  make  and  indicate  the  areas  in  which  he  may 
require  inputs  from  professional  individuals. 


BLOCK  1.4:  ANALYZE  EXISTING  COURSES 


Introduction 

A basic  objective  of  the  Interservice  Procedures  for  Instructional 
Systems  Development  is  to  facilitate  and  encourage  interservice  train- 
ing in  all  those  situations  which  meet  the  estabi ! shed  criteria.  3y 
careful  analysis  of  the  courses  existing  within  and  among  the  services 
according  to  a recommended  set  of  procedures,  It  is  expected  that  the 
benefits  of  interservice  training  are  more  likely  to  be  realized. 

These  procedures  provide  guidelines  for  making  the  analysis  and  sug- 
gest criteria  which  can  be  applied  in  order  to  estimate  the  potential 
usefulness  of  existing  training. 

Rationale 

These  procedures  are  included  at  this  point  in  the  model  to  avoid 
unnecessary  duplication  'of  effort  wherever  possible.  To  make  this 
analysis  in  the  absence  of  reliable  job  analysis  data  could  well  be 
misleading.  Many  courses  of  the  same  name  do  not  have  the  same  con- 
tent, nor  do  they  emphasize  the  same  tasks  or  conditions.  The  primary 
purpose,  then,  of  this  block  is  to  determine  the  degree  to  which  an 
existing  course  teaches  the  same  tasks  that  the  command  needs  to  teach. 

A second  important  reason  for  these  procedures  is  to  enable  any 
service  to  take  advantage  of  prior  work  done  within  that  service  or 
within  the  other  services.  The  position  taken  here  is  that  dupli- 
cation of  instruction  is  not,  in  and  of  itself,  evidence  of  bad 
planning.  In  many  cases,  the  location  and  existence  of  a course  which 
is  operating  at  capacity  may  well  preclude  sending  additional  people 
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to  It.  Certain  obvious  examples,  such  as  basic  training,  provide 
the  basis  for  these  conclusions. 

Probably  the  most  important  reason  for  the  inclusion  of  this 
specific  step  is  to  allow  for  the  orderly  communication  about  and 
the  common  acceptance  of  courses  which  are  selected  for  joint  develop- 
ment. If  a course  to  be  jointly  developed  by  two  or  more  services  is 
developed  according  to  the  same  procedures  and  techniques,  it  is  far 
more  likely  to  meet  the  requirements  of  all  the  participants.  It  is 
to  that  end,  that  these  procedures  have  been  designed. 

Inputs 

The  inputs  to  Block  1.4  include  any  existing  job  analyses,  tasks 
selected  for  training,  or  job  performance  measures  which  have  been 
developed.  These  existing  materials  can  come  from  two  separate  sources: 

1.  from  existing  courses  currently  in  operation  in  one 
or  more  of  the  services,  and 

2.  from  new  developments  where  job  analyses,  task  selection, 
or  job  performance  measures  have  been  produced. 

It  is  more  likely  that  the  analysis  of  existing  courses  would  take 
place  within  a service  although  many  opportunities  for  sharing  train- 
ing do  exist.  It  is  more  likely  that  new  front-end  work  will  have  been 
the  product  of  joint  development  efforts. 

Procedures 

There  are  two  separate  kinds  of  procedures  involved,  ine  first 
involves  locating  the  existing  courses  and  obtaining  sufficient  in- 
formation to  make  a careful  analysis.  The  second  involves  the  analysis 
itself. 
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Locating  existing  courses  is  ordinarily  done  best  through  a search 
of  the  published  documents  available  from  the  various  services.  These 
documents  are  referenced  In  the  manuals.  The  primary  effort  is  to 
locate  the  organization  which  h«s  proponency  for  the  course  in  ques- 
tion in  each  service  and  to  communicate  with  them  according  to  estab- 
lished procedures.  Inquiries  of  this  nature  must  become  routine  in 
order  to  avoid  potential  conflicts  over  perceived  vested  Interests. 

What  is  particularly  needed  at  this  point  is  not  the  actual  instruc- 
tional materials,  but  the  documentation  of  the  implementation  and  outputs 
of  job  analysis,  selection  of  tasks  for  training,  and  development  of 
job  performance  measures. 

Having  obtained  this  information,  the  next  procedures  involve  the 
analysis  of  the  first  three  steps  in  ISD,  the  job  analysis,  task  se- 
lection, and  performance  measure  functions.  If  the  job  analysis  has 
been  accomplished  by  an  acceptable  procedure,  and  the  survey  sources 
were  similar  to  those  for  which  the  current  training  need  exists,  the 
task  analysis  is  acceptable  as  a reliable  basis  for  training.  The 
criteria  for  selection  of  tasks  for  training  must  be  carefully  re- 
viewed. It  has  been  found  that  confusion  often  exists  on  such  questions 
as  "operation  of  equipment"  as  opposed  to  "maintenance  and  repair  of 
equipment."  Tasks  involved  in  such  functions  must  be  carefully  exam- 
ined to  be  sure  that  the  criteria  used  for  selection  of  tasks  for 
training  are  acceptable  for  the  current  purpose. 

The  third  important  point  of  analysis  is  the  job  performance  meas- 
ures in  the  existing  course.  The  ideal  is  to  find  a perfect  match 
between  the  tasks  selected  for  training  and  the  measures  of  those  tasks. 
If  the  job  analysis  has  been  done  correctly,  the  tasks  selected  for 
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training  appear  to  meet  the  needs  of  the  current  course  planning 
effort,  and  the  job  performance  measures  appear  to  have  been  carefully 
constructed  and  validated,  the  opportunity  for  using  the  existing 
course  Increases  greatly. 

The  final  major  part  of  the  analysis  Is  to  determine  if  the 
existing  course  has  been  properly  validated;  that  Is,  has  it  been 
determined  that  students  who  take  the  course  can,  as  a result  of 
taking  the  course,  pass  the  job  performance  measures.  If  the  course 
has  not  been  validated,  this  must  be  done  before  the  course  can  be 
accepted  as  suitable  for  the  training  needs. 

Outputs 

The  output  for  Analyze  Existing  Courses  includes  a rationale  for 
the  analysis  and  the  procedures  that  were  followed  in  making  the 
analysis.  It  should  be  complete  enough  to  indicate  that  an  adequate 
search  was  made  and  that  the  analysis  met  the  current  needs.  A 
second  output  includes  a summary  listing  of  the  courses  analyzed  and 
those  sources  examined  to  discover  the  courses.  The  third  element 
of  the  output  includes  the  major  decisions  made  following  the 
analysis  including  the  rationale  and  explanation  of  the  decisions. 
Ordinarily,  this  decision  will  not  be  a simple  yes  or  no,  but  will  be 
somewhat  more  fragmented.  It  Is  more  likely  that  only  parts  of 
existing  instruction  may  be  used  as  part  of  the  new  instruction. 

Management  Decisions 

Of  all  the  management  decisions  required  in  Phase  I,  the  decision 
to  use  all  or  part  of  an  existing  course  can  be  one  of  the  more  im- 
portant. Opportunities  for  savings  in  resources  and  time  are  clearly 
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apparent  If  suitable  courses  already  exist.  On  the  other  hand,  using 
Instruction  simply  because  it  exists  may  not  be  a good  decision  for 
a manager  to  make. 

The  manager  should  be  careful  to  avoid  the  normal  kinds  of  error 
In  these  decisions,  namely,  to  accept  courses  which  do  not  meet  the 
needs  or  to  reject  courses  which  do  meet  the  needs,  because  they 
were  not  developed  locally. 

These  decisions  must  be  based  on  much  more  than  the  existence  or 
non-existence  of  acceptable  instruction.  They  must  take  into  account 
the  future  needs  of  the  service,  the  ability  of  existing  Instruction 
to  accept  new  enrollees,  whether  or  not  existing  instruction  can  be 
exported  to  other  locations  for  delivery,  and  whether  the  opportun- 
ities for  true  cost  savings,  increased  effectiveness,  or  decreases 
in  time  are  predictable. 

Fortunately,  such  categorical  decisions  as  "accept"  or  "reject" 
rarely  are  required.  More  often,  it  is  the  function  of  the  manager 
to  decide  which  segments  or  subsets  of  existing  instruction  can  be 
used  and  which  can  not. 

Acquiring  expertise  in  the  cooperative  course  development  effort 
within  the  services  presents  the  opportunity  to  obtain  one  of  the 
greater  benefits  in  the  entire  ISD  process.  It  should  also  be  realized 
that  cooperative  development  efforts  have  the  firm  support  of  the 
major  commands,  the  congress,  and  central  military  agencies.  Using 
these  procedures  competently  can  improve  the  opportunity  for  recommend- 
ing interservice  training  "from  the  bottom  up"  rather  than  having  it 
imposed  "from  the  top  down." 


BLOCK  1.5:  SELECT  INSTRUCTIONAL  SETTINGS 


Introduction  .t  « 

' ’ v. 

To  Insure  that  adequate  training, resources  wilt  be  available  when 

required  upon  the  completion  of  the  development  of  training,  an 
early  decision  must  be  made  assigning  the  tasks  to  the  instructional 
setting  or  settings  In  which  they  will  be  trained.  The  optimal  in- 

y 

structional  setting  for  tasks  is  one  that  provides  the  most  effective 
and  efficient  training  to  those  members  of  the  DOS  who  require  the 
training.  Setting  decisions  are  very  important  in  arranging  train- 
ing so  that  the  trainee  will  be  trained  at  the  point  in  his  career 
or  assignment  when  he  needs  the  knowledge  or  skills. 

There  are  five  instructional  settings  in  the  ISD  procedures  to 
which  tasks  can  be  assigned: 

1.  Job  Performance  Aids  (JPAs)  (even  though  a\iob  per- 

« 

formance  aid  is  not  technically  instruction,  it  can 
serve  to  eliminate  the  need  for  instruction) 

2.  Self-Teaching  Exportable  Packages  (STEPs) 

3.  Formal  On-The-Job  Training  (FOJT) 

4.  Installation  Support  Schools  (ISS) 

5.  Resident  Schools  (RS) 

Guidelines  have  been  established  for  the„assignment  of  tasks 
to  instructional  settings  and  these  guidelines  are  determined  in  part 
by  the  capability  and  resources  of  the  various  settings  as  well  as 
the  need  for  training  at  any  point  in  time.  The  guidelines  can  be 
applied  to  the  various  clusters  of  similar  tasks  which  can  be  nominated 
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for  specific  settings.  After  all  of  the  tasks  have  been  accounted  for, 
the  total  list  of  tasks,  with  the  training  setting  nominations  are 
then  discussed  with  appropriate  management  personnel.  Trade-offs  are 
made  based  on  the  requirements  and  resources  of  the  command  in  order 
to  arrive  at  the  most  effective,  time-efficient,  and  economical 
assignments. 

Rati onale 

Setting  decisions  have  a direct  impact  on  the  total  training  sys- 
tem. This  is  particularly  true  when  tasks  have  been  assigned  orig- 
inally to  one  setting  and  then,  on  the  basis  of  sound  application 
of  ISO  guidelines,  are  assigned  to  another  setting.  As  a consequence, 
personnel  often  must  be  reassigned  to  the  new  setting  to  provide  the 

• . V 

instruction.  ■'  " 

' l ' 

If  setting  selection  guidelines  are  not  followed,  newer  training 

approaches  are  difficult  to  introduce.  Further,  some  settings  are 
less  costly  and  there  is  good  reason  to  believe  that  some  settings 
will  definitely  prove  to  be  more  cost-effective  as  sufficient  data 
becomes  available  to  make  these  evaluations. 

if  all  Instruction  is  developed  according  to  ISD  principles,  it 
will  no  longer  be  possible  to  claim  that  some  settings  must  always 
be  second-best.  As  settings  become  equally  effective,  the  decision 
about  selection  will  become  easier  to  make. 

Inputs 

The  Inputs  to  Select  Training  Settings  are: 

All  of  the  documentation  from  Blocks  1.1,  1.2,  1.3,  and  1.4. 


II 
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Procedures 

The  procedures  to  be  followed  for  assigning  tasks  to  instructional 
settings  Involve  the  application  of  guidelines  for  determining  the 
suitability  of  each  setting  for  the  particular  tasks  in  question.  After 
the  tasks  have  been  clustered  according  to  the  skill  level  of  personnel 
who  will  be  Involved,  they  are  then  clustered  according  to  the  con- 
straints such  as  resources,  equipment,  and  facilities  which  are  required 
to  train  the  tasks.  After  the  initial  clustering,  the  tasks  are  ex- 
amined first  to  see  whether  they  are  suitable  for  treatment  by  a job 
performance  aid.  Job  performance  aids  range  from  simple  lists  of 
Instructions  or  decals  applied  directly  to  equipment  to  very  complex, 
fully  procedural i zed  aids  for  maintenance  and  repair  tasks  which 
contain  minute  step-by-step  procedures  for  task  performance. 

Tasks  can  be  assigned  directly  to  a job  performance  aid  when  task 
performance  can  be  totally  accomplished  using  the  JPA.  Often,  JPAs 
are  suitable  for  a task,  but  some  training  must  be  given  on  the  use 
of  the  job  performance  aids. 

If  job  performance  aids  are  not  satisfactory,  the  second  setting 
considered  is  the  self-teaching  exportable  package  (STEP) . STEPs  can 
he  any  form  of  exportable  instruction  which  does  not  require  the 
presence  of  an  instructor  at  the  training  setting.  However,  feedback 
must  be  provided  either  in  the  form  of  correct  answers  or  school  solu- 
tions or  in  the  form  of  responses  from  correspondence  course  instruc- 
tors, STEPs  can  be  in  print  form,  in  audiovisual  form,  or  in  the  form 
of  a kit  which  can  be  assembled  or  manipulated. 

A third  training  setting  which  is  given  careful  consideration  is 

. 1 

formal  on-the-job  training  (FOJT).  If  resources  on  the  job  permit  'i 
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the  assignment  of  tasks  to  FOJT,  and  a management  system  is  in  place 
which  can  handle  the  administration  and  testing  involved  in  FOJT,  tasks 
can  be  considered  for  assignment  to  this  setting.  FOJT  has  the  advantage 
of  providing  continuous  training  on  tasks  which  are  of  immediate  need  to 
the  trainee.  Further,  FOJT  can  continue  for  whatever  length  of  time  is 
necessary  for  the  trainee  to  achieve  mastery  of  the  test.  FOJT  is  limited 
to  those  situations  where  it  is  administratively  possible  to  conduct  the 

t 

training,  and  where  the  facilities  are  adequate. 

If  a task  is  not  assigned  to  any  of  the  above  settings,  it  might  meet 
the  criteria  for  assignment  to  an  Installation  Support  School  (ISS).  In- 
stallation support  schools  are  operated  by  local  commands,  principally  to 
meet  local  needs  either  in  those  DOSs  where  shortage  of  personnel  exists, 
where  new  equipment  or  procedures  are  required  in  the  field,  or  where 
there  is  only  a temporary  need  for  the  training. 

Finally,  if  none  of  the  other  settings  prove  to  be  satisfactory,  the 
task  can  be  considered  for  resident  school  instruction.  Since  resident 
school  instruction  is  ordinarily  believed  to  be  the  most  expensive  of  the 
settings,  it  should  be  used  only  when  the  other  forms  of  training  will  not 
suffice.  There  are  certainly  many  excellent  reasons  why  one  would  choose 
to  use  resident  school  instruction  rather  than  other  bettings.  But,  resi- 
dent schools  do  have  limitations,  particularly  in  being  able  to  train  tasks 
for  which  the  individual  has  an  immediate  need.  Further,  often  individuals 
who  are  graduates  of  resident  schools  are  not  assigned  to  perform  the  tasks 
immediately  upon  reaching  their  duty  station  and  the  trainees  lose  pro- 
ficiency on  specific  tasks  while  they  are  performing  other  duties.  Because 
the  resident  school  cannot  often  be  exactly  like  the  real  world,  some 
training  does  not  transfer  well  to  the  job. 
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Then,  after  all  of  the  nominations  are  completed,  the  persons 
responsible  for  the  setting  being  considered  meet  to  discuss  the 
necessary  trade-offs  required  to  achieve  the  desired  training  re- 
sults. Appropriate  management  personnel  familiar  with  the  altern- 
ative costs  of  training  settings  participate  in  these  discussions  so 
that  resources,  costs,  and  requirements  can  all  be  given  adequate 
considerati on. 

Outputs 

The  outputs  of  the  Select  Instructional  Settings  block  include 
the  following: 

1.  The  list  of  tasks  selected  for  training  along 
with  their  assignment  to  an  appropriate  training 
setting. 

2.  Documentation  and  explanation  for  the  rationale  of 
the  choices. 

Management  Decisions 

Since  the  management  decisions  in  Phase  I have  profound  effects 
throughout  the  remainder  of  the  instructional  design  and  development 
effort,  every  effort  should  be  made  to  ensure  that  the  best  possible 
data  are  furnished  to  the  decision  maker. 

Decisions  involved  in  this  block  will  include  resources,  costs, 
and  the  coordination  of  departments  within  the  command  and  also  with 
many  persons  external  to  the  command. 

Training  setting  decisions  will,  through  time,  become  more  and 
more  dependent  on  adequate  and  accurate  cost  information.  Two  specific 


approaches  should  be  studied  now,  and  future  developments  In  train- 
ing setting  costs  should  be  carefully  followed.  The  Navy  has  de- 
veloped a cost  model , A Technique  for  Choosing  Cost-Effective 
Instructional  Delivery  Systems  (TECEP),  which  is  a straightforward 
procedure  for  beginning  the  systematic  collection  of  cost  data. 

A second  and  more  specific  approach,  is  contained  in  a study  by 
the  Rand  Corporation. 

References 
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for  Choosing  Cost-Effective  Instructional  Delivery  Systems  {TAEG 
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PHASE  II: 


DESIGN 


Preceding  page  blank 


BLOCK  II. 1:  DEVELOP  OBJECTIVES 

Introduction 

In  this  block,  using  the  outputs  from  Phase  1,  the  learning  tasks 
are  analyzed.  It  is  in  this  block  that  the  break  occurs  between  the 
job  world  and  the  training  world.  It  has  long  been  recognized  that 
there  is  a distinct  difference  between  the  performance  of  a task 
under  job  conditions  and  learning  how  to  perform  that  task  in  an  in- 
structional setting. 

The  job  performance  measures  (JPMs)  developed  in  Block  1.3  provide 
the  basis  for  developing  learning  objectives.  The  learning  objectives 
developed  in  this  block  are  the  common  three  part  objectives  that 
have  been  used  for  a long  time  involving  a statement  of  the  action. 


conditions,  and  standards  involved  in  the  performance.  In  the  ISD 
model,  learning  objectives  are  written  for  four  distinct  types  of 
learning:  information,  mental  skills,  physical  skills,  and  attitudes. 

This  is  because  it  has  been  determined  that  the  conditions  for  learn- 
ing in  each  of  these  categories  is  different.  Proper  classification 


of  learning  into  the  categories  allows  for  the  design  of  instruction 
to  meet  specific  learning  conditions. 

The  terminal  learning  objectives  are  analyzed  in  sufficient  detail 
so  that  all  of  the  knowledges  and  skills  necessary  to  learn  the  ob- 
jective are  specified.  When  all  of  the  learning  objectives  have  beer, 
prepared,  they  are  ready  to  have  test  items  written.  This  is  done  in 
the  next  block. 
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Rationale 

Learning  objectives  have  been  found  to  be  an  extretnely  good  means 
for  controlling  the  Intent  of  instruction.  They  communicate  adequately 
to  the  student,  the  instructor  and  the  instructional  designer  exactly 
what  it  is  the  student  is  expected  to  do  in  order  to  demonstrate  his 
mastery  of  the  objective. 

Learning  objectives  serve  as  a control  over  the  content  of  instruc- 
tion since  they  must  be  based  directly  on  the  job  performance  measures. 
Because  there  is  a direct  relationship  between  each  learning  objective 
and  a job  performance  measure  or  part  of  a job  performance  measure, 
much  unnecessary  instruction  can  be  eliminated. 

For  many  types  of  students,  simply  providing  the  learning  ob- 
jectives and  the  necessary  learning  resources  is  all  that  is  required 
to  obtain  acceptable  performance.  Further,  since  learning  objectives 
are  stated  in  terms  of  what  the  student  is  expected  to  do  rather  than 
what  the  Instructor  or  supervisor  is  expected  to  do,  a much  better  con- 
trol over  the  output  of  instruction  can  be  obtained. 

Inputs 

The  inputs  to  Block  II. 1,  DEVELOP  OBJECTIVES,  consist  of  the  total 
output  from  Phase  I.  Each  specific  Phase  I block's  output  can  help 
to  clarify  or  further  specify  the  necessary  instruction  which  the  de- 
signer can  use  to  develop  learning  objectives. 

Procedures 

The  first  step  in  developing  learning  objectives  is  to  prepare 
terminal  learning  objectives  which  are  the  direct  translations  of  the 
job  performance  measures  into  learning  objectives  for  the  training  world. 
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These  terminal  learning  objectives  will  describe  the  actions,  con- 
ditions and  standards  which  will  be  met  in  the  training  world  in 
order  to  prepare  the  person  for  the  job.  A terminal  learning  ob- 
jective is  a specific  description  of  the  action  the  learner  is  to 
exhibit  after  training,  the  conditions  under  which  the  action  will 
take  place  and  the  standards  or  criterion  which  must  be  reached  for 
satisfactory  performance. 

Terminal  learning  objectives  require  action  statements  which 
have  been  developed  using  verbs  which  call  for  observable  behaviors 
on  the  part  of  the  student.  The  action  is  a statement  of  student 
behavior,  describes  the  creation  of  a product,  or  some  other  act 
which  can  be  accepted  as  evidence  that  the  intended  outcome  has 
occured.  In  order  to  be  acceptable,  this  behavior  must  be  observ- 
able and  subject  to  evaluation.  The  conditions  portion  cf  the  learn- 
ing objective  includes  specific  statements  about  what  the  student  is 
given  (books,  training  manuals,  etc.).  The  standards  portion  of  the 
learning  objectives  includes  the  criteria  for  acceptable  performance. 
These  criteria  are  statements  about  completeness,  accuracy,  available 
time,  etc. 

Information  type  learning  objectives  require  the  student  to  recall 
bodies  of  knowledge.  That  is,  the  student  may  be  asked  to  recall  in- 
formation necessary  to  complete  a form.  Mental  skills  type  learning 
objectives  require  the  individual  to  Identify,  classify,  use  rules, 
or  solve  problems  which  involve  thinking,  creating,  and  analyzing. 
Physical  skills  type  learning  objectives  Involve  some  physical  or 
manipulative  activities.  They  require  movement  of  some  of  the  muscles 
of  the  body  and  are  directly  observable.  Attitude  type  learning 
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objectives  usually  are  not  directly  observable  but  are  ordinarily 
reflected  in  the  choices  that  an  Individual  makes.  For  example,  if 
an  individual  had  a choice  between  two  different  methods  tf  per- 
forming a task,  one  safe  and  one  unsafe,  and  he  chose  the  safe  al- 
ternative, he  could  be  said  to  have  a positive  attitude  toward  working 
safely. 

In  the  area  of  mental  skills,  it  is  often  the  case  that  an  indi- 
vidual cannot  learn  to  do  one  skill  if  he  has  not  mastered  certain 
prerequisite  skills.  Mental  skills  learning  objectives  must  be 
analyzed  for  the  required  prerequisite  behavior  until  a level  of 
skills  is  reached  at  which  it  is  assumed  that  all  trainees  have 
already  reached  mastery. 

For  example  (see  Figure  Z\  if  a terminal  learning  objective  is  for 
an  individual  to  navigate  from  point  A to  point  E,  it  would  be  neces- 
sary to  analyze  the  prerequisite  skills.  In  this  case  it  would  be 
necessary  for  the  individual  to  interpret  symbols  on  a map  and  to 
use  a compass.  Further,  in  order  to  Interpret  the  map,  he  must  be  able 
to  use  the  legend  and  to  use  the  grid  system.  In  order  to  use  the 
legend  he  must  be  able  to  match  symbols  to  actual  terrain  and  match 
the  symbols  to  the  legend.  Prior  to  matching  the  symbols,  he  must 
be  able  to  identify  the  symbols. 

This  process  of  analysis  details  every  step  to  be  sure  that  nothing 
has  been  overlooked  so  that  individuals  are  not  given  instruction  for 
which  they  are  not  prepared,  and  at  the  same  time,  are  not  given  instruc- 
tion which  they  have  already  mastered,  (see  Figure  2) 
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FIGURE  2:  Learning  Analysis 

Outputs 

The  outputs  from  Block  11,1  include  a list  of  terminal  learning 
objectives,  learning  obiectives,  and  learning  steps  analyzed  to  the 
level  where  it  is  assumed  ’■.hat  individuals  beginning  instruction  will 
have  the  necessary  prerequisites.  These  objectives  will  be  used  for 
the  development  of  tests  in  Block  1 1. 2 and  for  the  sequencing  of  ob- 
jectives in  Block  II. 4.  In  Block  II. 2,  assumptions  about  the  level 
of  prerequisite  knowledge  and  skill  in  the  target  population  will 
be  verified. 
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Management  Decisions 

Management  decisions  In  Block  II. 1 will  involve  principally  the 
inspection  of  the  learning  analyses  and  the  learning  objectives  de- 
veloped. The  learning  objectives  should  be  properly  stated,  analyzed, 
and  organized.  The  manager  must  look  for  those  objectives  which  do 
not  conform.  There  are  many  potential  errors  in  the  development  of 
objectives  including: 

1.  The  student's  mental  process  rather  than  observable 
behavior. 

2.  The  standard  is  absent  or  deficient. 

3.  The  statement  of  conditions  is  missing  or  poorly 
specified. 

4.  The  objective  is  stated  in  instructor  rather  than 
learner  terms. 

5.  The  behavior  stated  is  too  broad.  It  does  not  ade- 
quately describe  the  specific  outcomes. 

In  reviewing  the  learning  objectives,  the  manager  must  identify 
the  relationship  between  the  terminal  learning  objective  as  stated  and 
the  job  performance  measure  upon  which  it  is  based.  Learning  objectives 
should  be  examined  to  Insure  that  each  subordinate  step  is  directly 
related  to  the  terminal  performance.  Often,  learning  objectives  are 
written  in  such  a way  as  to  preserve  existing  instruction  rather  than 
to  conform  to  the  requirements  of  the  job  performance  measure.  It  is 
the  manager's  job  to  insure  that  these  relationships  are  direct  and 
clear. 


BLOCK  II. 2:  DEVELOP  TESTS 

Introduction 

In  Block  I 1.1,  the  job  performance  measures  were  analyzed  and 
learning  objectives  were  written  which  described  what  had  to  be 
learned  In  order  for  the  task  to  be  performed.  Often  what  has  to  be 
learned  takes  more  time  and  has  more  Intermediate  steps  than  task 
performance  itself. 

In  this  block,  tests  are  written  which  measured  the  outcomes  of 
instruction  at  whatever  level  of  detail  the  instruction  is  offered. 

Tost  items  are  written  for  each  kind  of  learning  objective, 
making  sure  that  all  significant  features  of  what  has  to  be  learned 
are  represented  in  the  test.  These  test  items  may  be  used  on  entry 
tests,  pretests,  unit  posttest,  or  end  of  course  tests. 

Enough  test  items  are  written  on  all  of  the  learning  objectives 
to  allow  for-  initial  testing  of  the  learning  objectives  and  enough 
alternative  forms  of  the  test  to  permit  retesting  those  situations 
where  this  is  necessary.  Once  the  test  items  have  been  developed, 
they  are  ready  for  use  in  the  next  step,  DE5CRIBE  ENTRY  BEHAVIOR,  in 
which  assumptions  about  the  student  population  are  verified  and  modified. 

Rationale 

The  key  to  any  successful  instructional  program  is  the  precision 
with  which  what  is  taught  is  tested.  The  quality  of  the  tests  developed 
here  will  have  an.  important  impact  on  the  quality  of  the  instruction. 

The  tests  must  have  good  technical  characteristics  (and  good  reliability 
and  validity)  since  they  will  form  the  basis  of  many  decisions  that  will 

be  made  about  students  and  about  the  quality  of  the  Instruction. 
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Inputs 

The  Inputs  to  the  Develop  Tests  block  Include: 

1.  job  performance  measures  developed  in  Block  1.3, 

2.  the  setting  selection  Information  from  Block  1.5,  and 

3.  all  learning  objectives  developed  in  Block  II. 1. 

Procedures 

The  procedures  used  in  this  block  resemble  those  used  in  Block  1.3, 
CONSTRUCT  JOB  PERFORMANCE  MEASURES.  The  principal  differences  in 
the  procedures  used  include: 

1.  Tests  required  to  measure  learning  objectives  and 
learning  steps  are  prepared  in  much  greater  detail  than 
those  required  to  test  job  performance.  Tests  prepared 
to  measure  the  effects  of  instruction  must  be  written  su 
they  can  also  be  used  for  diagnostic  purposes. 

2.  Many  items  are  written  on  knowledge  and  skill  subordinate 
to  the  terminal  learning  objective.  These  items  are  not 
important  in  and  of  themselves  but  only  ar.  they  allow  the 
testing  of  the  subordinate  skills. 

3.  Tests  for  different  learning  objectives  usually  appear 
as  different  items.  That  is,  tests  for  mental  skills 
generally  require  the  application  of  the  skill  while  tests 
of  information  ordinarily  require  the  individual  simply 

to  recall  the  information. 

Performance  standards  are  usually  less  demanding  early  in  training 
than  they  are  later  in  training.  In  many  cases,  test  items  need  to 
be  written  which  reflect  the  level  of  skill  necessary  at  an  early  point 
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of  training  and  a different  test  Item  used  at  a later  stage  In  train- 
ing in  order  to  more  closely  approximate  real  world  job  performance. 

If  existing  courses  are  available,  the  test  items  developed  in  this 
block  should  be  administered  to  present  or  recent  graduates  of  the 
existing  course  In  order  to  try  out  the  test  Item  and  also  to  define 
the  baseline  content  coverage  of  the  existing  instruction. 

Finally,  the  test  Items  are  revised  based  on  the  initial  tryout 
to  eliminate  any  inconsistencies,  confusing  points,  or  difficulties 
in  performance. 

Outputs 

The  outputs  for  Block  II. 2 include: 

1.  items  for  entry  tests, 

2.  items  for  pretests, 

3.  items  for  within  course  test,  and 

4.  items  for  posttests. 

Management  Decisions 

Many  of  the  problems  encountered  in  developing  tests  have  solu- 
tions of  a technical  nature  which  often  require  little  management 
intervention.  The  questions  such  as  improving  the  reliability  or 
validity  of  the  tests  often  have  relatively  straightforward  methodology. 

The  manager  will  need  to  be  concerned  with  the  degree  of  fidelity 
available  in  the  testing  situation,  the  completeness  and  thoroughness 
of  the  tests,  and  the  use  of  these  tests  in  establishing  the  baselines. 

The  manager  will  also  be  concerned  with  the  establishment  of  cri- 


teria or  cut-off  scores  which  are  used  in  many  situations  to  separate 
people  into  two  groups:  go  or  no-go.  Since  many  of  the  criteria 
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currently  ussd  are  arbitrary,  the  manager  must  be  concerned  with 
establishing  attainable  arbitrary  cut-off  scores  which  have  some  re 
lationshlp  to  the  .job  world.  Since  Instruction  Is  not  offered  for 
Its  own  sake,  but  as  a means  to  an  and,  the  manager  must  be  con- 
cerned with  the  kinds  of  test  Items  used  to  be  sure  that  they  are 
consistent  with  the  overall  alms  of  the  Instructional  program. 


BLOCK  II. 3:  DESCRIBE  ENTRY  BEHAVIOR 


Introduction 

Adequate  design  of  ISD  training  requires  a careful  analysis 
and  description  of  the  entry  behavior  of  the  trainee.  Entry  be- 
havior falls  in  two  principal  classes: 

1.  Basic  aptitude  and  ability,  and 

2.  Acquired  knowledge  and  skills. 

In  the  short  term,  very  little  can  be  done  to  change  the  basic 
aptitudes  and  ability  of  the  entering  trainee.  However,  longer  term 
results  can  suggest  the  need  for  different  selection  criteria. 

Assumptions  must  be  made  in  Block  II. 1 about  the  level  of  know- 
ledges and  skills  of  the  trainee.  In  this  block,  these  assumptions 
are  verified  or  adjusted  depending  on  the  result  of  testing  of 
entry  skills. 

In  addition  to  the  entry  test  which  is  used  to  adjust  the  beginning 
point  of  a course,  pretests  for  the  instructional  unit  are  developed 
to  see  to  what  extent  students  have  already  mastered  the  skills  to 
be  taught  in  the  course.  Provisions  can  be  made  for  students  to  by- 
pass certain  blocks  of  instruction  if  they  have  already  met  the  skills. 

Rationale 

A fundamental  concept  in  ISD  is  that  a maximum  proportion  of  qual- 
ified trainees  should  meet  the  training  requirements  with  the  instruc- 
tion provided.  Since  many  trainees  will  begin  the  instruction  with 
different  knowledge,  skill  and  ability,  it  is  necessary  to  document 
this  information  so  that  course  design  can  be  tailored  to  the  users. 
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Further,  In  those  cases  where  It  Is  desirable,  more  capable  trainees 
can  be  allowed  to  advance  through  the  instruction  faster  and  thereby 
reduce  the  average, training  time.  Occasionally  it  is  necessary  to  de- 
sign preliminary  instruction  to  je  offered  to  groups  of  trainees  who 
have  not  achieved  the  necessary  knowledge  and  skills  to  benefit  from 
the  Instruction. 

Inputs 

The  inputs  to  the  Describe  Entry  Behavior  block  are: 

1.  The  terminal  learning  objectives,  learning  objectives, 
and  learning  steps  developed  in  Block  II. 1. 

2.  The  test  items  for  each  of  the  learning  objectives  * 

developed  in  Block  1 1. 2. 

3.  Administrative  criteria  from  the  personnel  system. 

Procedures 

The  first  step  is  to  develop  an  entry  test  based  on  the  test  items 
developed  in  Block  1 1. 2.  These  test  items  were  prepared  as  a result 
of  a learning  analysis.  The  learning  analysis  was  continued  until  a 
level  was  reached  at  which  it  was  assumed  that  all  entering  trainees 
would  have  mastered  these  subordinate  skills. 

The  test  based  on  these  items  is  administered  to  a sample  of  stu- 
dents typical  of  the  population  for  which  the  course  was  Intended. 

The  tests  are  scored  and  used  to  verify  or  revise  the  assumptions 
made  about  the  entry  knowledge  and  skills  of  the  trainees.  Generally, 
all  items  passed  by  most  of  the  trainees  would  not  be  included  in  the 
instruction,  and  all  items  not  passed  by  a substantial  portion  of  the 
trainees  would  be  included. 
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This  test  will  often  detect  certain  areas  that  perhaps  as  many, 
as  25  percent  of  t.he  trainees  have  not  mastered  but  which  might  lend 
themselves  to  preliminary  training.  If  those  prerequisite  skills  can 
be  provided  by  brief  preliminary  training,  the  instruction  can  be  de- 
signed to  serve  the  level  of  the  majority  of  the  trainees. 

The  results  of  this  administration  of  the  test  can  be  used  to 
provide  for  further  analysis  in  Block  II. 1,  to  revise  the  test  items, 
or  to  verify  that  the  assumptions  made  in  Block  II. 1 were  correct. 
Hopefully  this  process  of  testing  and  revision  can  be  continued  until 
the  instruction  is  serving  the  largest  proportion  of  th^  entering 
trainees. 

For  instruction  in  which  it  is  possible  for  the  trainees  to 
bypass  certain  instruction  on  the  basis  of  their  performance  on  pre- 
tests of  units  of  that  instruction,  the  pretest  can  be  developed  in 
this  block.  If  the  course  is  self-paced,  the  student  will  be  able  to 
test  as  far  into  the  course  as  he  is  able  and  begin  instruction  on 
exactly  that  module  or  unit  that  matches  his  current  knowledge  and 
skill.  In  block  scheduled  courses,  it  is  often  possible  to  place 
students  ir.  more  advanced  classes  if  they  can  pass  the  pretest  over 
the  blocks  of  instruction. 

Outputs 

The  outputs  of  Block  II. 3 are: 

1.  The  entry  test  which  has  been  developed  to  verify  the 
assumptions  made  in  the  learning  analysis. 

2.  Pretests  for  those  units  of  instruction  that  students 
may  bypass. 
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3.  The  revised  learning  objectives  and  tests  which  match 
the  entry  behavior  of  the  students  as  verified  in  this 
block. 

Management  Decisions 

Management  decisions  in  this  block  center  around  the  question  of 
the  exact  level  of  Instruction  at  which  the  course  will  begin,  whether 
or  not  remedial  or  preliminary  instruction  will  be  given  to  some  of 
the  trainees,  and  what  the  required  entry  abilities  and  aptitudes  of 
trainees  should  be.  The  manager's  job  in  this  instance  is  to  use  the 
entry  test  and  pretest  data  to  match  as  precisely  as  possible  the  in- 
struction with  the  entry  behavior  of  the  trainees.  This  is  to  avoid 
training  individuals  on  materials  they  have  already  mastered  and  to 
avoid  offering  instruction  to  unprepared  individuals  when  they  can 
not  possibly  profit  from  that  Instruction. 

The  cost  implications  of  correct  decisions  in  this  block  are 
far-reaching.  All  instruction  which  can  be  eliminated  through  entry 
testing  does  not  have  to  be  developed.  Further,  any  preliminary  or 
remedial  instruction  which  has  to  be  offered  should  be  justified  on 
the  basis  that  it  serves  an  appropriate  proportion  of  trainees  and 
enables  the  course  as  a whole  to  function  more  efficiently. 


BLOCK  II. 4;  DETERMINE  SEQUENCE  AND  STRUCTURE 


Introduction 

The  specific  purpose  of  this  block  is  to  identify  those  terminal 
learning  objectives  (TLOs),  learning  objectives  (LOs),  and  learning 
steps  (LSs)  which  are  independent  of  each  other,  those  which  are 
dependent,  and  those  which  may  have  supportive  relationships.  When 
two  learning  objectives  (the  term  "learning  objectives"  is  used 
here  to  refer  to  TLOs,  LOs,  and  LSs.)  are  independent,  the  learning 
of  one  has  no  effect  on  the  learning  of  another.  When  two  learning 
objectives  are  dependent,  it  is  necessary  to  learn  one  prior  to 
learning  the  other.  In  that  case,  the  learning  of  the  latter  learning 
objective  was  dependent  upon  learning  which  occurred  in  the  first 
learning  objective.  The  third  possibility  exists  when  the  learning 
of  one  learning  objective  supports  or  facil itates  the  learning  of 
another,  but  the  order  in  which  they  are  learned  is  not  important. 

That  is,  the  learning  in  one  will  transfer  to  the  other  no  matter 
which  one  is  learned  first. 

Organizing  the  learning  objectives  into  these  three  categories 
will  assist  the  developers  of  instruction  *n  two  ways: 

1.  It  will  identify  those  learning  objectives  chat  must 
be  presented  before  other  learning  objectives. 

2.  It  will  provide  maximum  flexibility  where  the  relation- 
ships are  supportive  or  independent. 


Rationale 

One  of  the  more  important  discoveries  in  the  last  few  yei.rs  in  the 

area  of  learning  research  has  been  the  notion  of  a learning  hierarchy. 
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A hierarchy  refers  to  a learning  situation  in  which  it  can  be  clearly 
demonstrated  that  the  learning  of  an  objective  cannot  occur  until  some 
prior  learning  has  taken  place.  For  example,  learning  to  multiply 
three-digit  numbers  by  three-digit  numbers  cannot  be  expected  to  occur 
until  a person  has  mastered  addition,  further,  learning  to  divide 
multi-digit  numbers  by  other  multi-digit  numbers  cannot  be  expected 
to  occur  until  the  individual  has  mastered  multiplication  and  sub- 
traction. Many  learning  relationships  exist  1 r.  which  it  is  necessary 
to  analyze  the  objectives  until  the  most  fundamental  learning 
steps  have  been  specified. 

Learning  objectives  are  independent  of  each  other  when  the  learn- 
ing of  one  of  them  has  no  measurable  effect  on  the  learning  of  the 
other.  For  example,  the  learning  of  maintenance  techniques  on  50 
calibre  machine  guns  would  have  little  if  any  impact  on  learning  how 
to  typewrite.  In  these  independent  relations  in' ps,  it  does  not  matter 
at  all  in  what  sequence  one  masters  the  learning  objectives. 

In  the  third  situation,  one  learning  objective  facilitates  the 
learning  of  a second  because  the  two  learning  objectives  have  psycho- 
logically commc'  properties.  In  this  situation,  the  learning  of 
one  learning  objective  would  make  the  learning  of  the  second  easier, 
but  it  would  "ot  matter  which  of  the  two  the  individual  learned 
first.  For  example,  operators  of  motor  vehicles  a*-e  ordinarily  taught 
to  opiate  simple  1 i ght-wei gh t equipment  before  they  are  allowed  to 
progress  to  complex  heavier  equipment.  In  this  car®,  for  learning  to 
occur,  it  is  not  necessary  to  learn  to  oprrate  the  light-weight 
equipment  first.,  it  simply  facilitates  learning  to  operate  the  heavy 
equipment,  and  is  less  costly. 
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It  is  necessary  to  demonstrate  the  facultative  effects  of  prior 
learning  on  subsequent  learning  In  order  to  be  sure  that  the  facul- 
tative arrangement  worked.  In  many  cases,  certain  materials  are 
taught  before  other  materials  solely  as  an  administrative  convenience 
or  for  historical  reasons  when  there  is  no  demonstrable  reason  based 
on  learning  efficiency. 

In  this  block,  when  dependence  is  determined  the  learning  objec- 
tives are  sequenced.  When  support  is  determined,  the  learning  ob- 
jectives are  kept  together  but  not  sequenced.  Independent  objectives 
will  be  sequenced  by  the  instructional  developers. 

Inputs 

The  inputs  to  Block  II. 4 Include  the  terminal  learning  objectives, 
the  learning  objectives,  and  the  learning  steps  as  analyzed  and  writ- 
ten from  Block  II. 1.  Further,  the  tests  from  Block  II. 2 which  go  with 
the  learning  objectives  are  included.  The  output  fro,,.  Block  II. 3, 

ANALYZE  ENTRY  BEHAVIOR,  is  also  available  for  examination  and  use. 

Procedures 

The  procedures  for  determining  whether  objectives  are  independent,  depen- 
dent, or  supportive  include  essentially  the  same  process  of  analysis  outlined 
in  Block  II. 1.  The  function  of  the  block  is  to  serve  mainly  as  a check  and 
organizer  of  materials  generated  in  the  prior  three  blocks.  It  also  pro- 
vides for  an  Independent  look  at  the  materials  to  be  sure  that  the  learning 
objectives  are  properly  classified  as  dependent,  independent,  or  supportive. 

Outputs 

The  output  of  Block  II. 4 includes  objectives  which  have  been  put 
into  three  categories,  independent,  dependent,  or  supportive,  and 
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sequenced  where  applicable,  in  addition,  the  learning  objectives  are 
structured  into  related  groups  to  facilitate  further  handling. 

Management  Decisions 

It  is  not  Intended  that  the  output  of  this  block  unnecessarily 
restrict  the  Instructional  designers  or  the  instructors  who  will  pre- 
sent the  material.  It  is,  therefore,  the  principal  duty  of  the  manager 
to  Insure  that  the  dependent  relationships  specified  as  output  are, 
in  fact,  dependent.  The  error  that  could  occur  here  is  to  claim 
that  two  relationships  are  dependent  when  they  are  merely  supportive. 
While  it  may  not  be  serious,  claiming  dependence  will  clearly  indicate 
order  of  instruction  and  that  could  be  unnecessarily  expensive, 

A second  error  that  could  occur  is  claiming  that  two  relationships 
are  independent,  when  in  fact  they  are  dependent.  In  those  situations, 
instruction  could  be  designed  out  of  order  so  that  individuals  have 
great  trouble  in  learning  what  is  expected. 

The  only  logical  errors  that  can  be  made  in  classifying  supportive 
relationships  «re  simply  failing  to  identify  those  relationships  which 
could  be  supportive.  Losing  this  proper  identification  could  make 
instruction  less  efficient  than  it  would  have  been  had  the  classifi- 
cation been  proper. 


BLOCK  III.l:  SPECIFY  LEARNING  EVENTS/ACTIVITIES 
Introduction 

Specific  learning  events/actlvitifcs  must  occur  in  the  Instructional 
environment  In  order  to  produce  the  desired  learning  outcomes.  In 
order  to  Identify  the  learning  events  and  activities  which  must  occur, 
two  kinds  of  learning  guidelines  are  used:  those  which  are  applicable 
to  a wide  variety  of  learning  objectives  and  those  which  are  unique 
to  each  category  of  learning.  The  objective  of  this  block  is  to  class- 
ify the  learning  objectives  into  appropriate  categories  and  to  identify 
the  learning  guidelines  necessary  for  optimum  learning  to  take  place. 

Further,  activities  are  identified  that  must  take  place  in  the 
instructional  environment  in  order  to  provide  training  most  directly 
related  to  task  performance.  Learning  guidelines  provide  the  basis 
for  the  instructional  developer  to  produce  materials  which  will  lead 
to  appropriate  learning  events,  and  subsequently  to  appropriate  activ- 
ities on  the  part  of  the  learner. 

These  learning  events  and  activities  are  specifically  taken  into 
account  in  the  following  block  when  the  media  selection  arid  instruc- 
tional management  plan  are  developed. 

Rationale 

The  learning  events  and  activities  are  dependent  upon  learning 
guidelines  which  have  been  developed  by  the  Training  Analysis  and 
Evaluation  Group.  The  guidelines  are  based  on  research  in  the  psychol- 
ogy of  learning  which  has  taken  place  during  the  last  50  years.  Much 
military  research  has  been  applied  directly  to  the  guidelines  and 
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they  represent  at  this  time  probably  the  most  complete  application 
of  existing  knowledge  in  that  area  of  learning  technology. 

It  has  been  known  for  a number  of  years  that  the  learning  of 
certain  kinds  of  objectives  occurs  in  fundamentally  different  ways 
than  does  the  learning  of  other  kinds  of  objectives.  Sometimes, 
regular  practice  is  required  in  order  to  maintain  a skill  while  at 
other  times  practice  in  performance  is  not  so  critical.  Further, 
it  has  been  found  that  providing  the  opportunity  for  practice  is 
often  more  effective  than  additional  reviews  of  the  material.  The 
learning  guidelines  therefore  specify  the  most  appropriate  approach 
for  initial  learning  and  maintenance  of  skills. 

Inputs 

The  inputs  to  the  Specify  Learning  Events/Activities  block  include: 

1.  The  learning  objectives  developed  in  Block  II. 1. 

2.  The  tests  developed  in  Block  1 1. 2. 

3.  The  entry  behavior  specifications  from  Block  II. 3. 

4.  The  sequence  and  structure  of  objectives  from  Block  II. 4. 

5.  All  the  outputs  from  Phase  I.  (available  for  reference) 

Procedures 

There  are  four  general  learning  guidelines  which  have  been  developed 
and  seem  to  apply  to  most  if  not  all  categories  of  learning.  These 
general  guidelines  are: 

1.  Inform  the  learner  of  the  objectives.  The  students  are 
told  exactly  what  is  expected  from  them,  exactly  what 
they  are  expected  to  do,  and  exactly  how  they  will  be 
tested  on  the  material  that  has  been  presented.  The 
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better  they  understand  what  is  expected,  the  more 
likely  they  will  be  able  to  perform  adequately. 

2.  Provide  for  active  practice.  Active  practice  is  a more 
specific  term  which  has  been  derived  from  the  more 
general  "performance  training"  and  "hands-on  training." 
Effective  training  requires  that  the  learner  be  given 
practice  in  the  performance  that  he  will  be  expected 

to  exhibit  at  the  end  of  training. 

3.  Provide  guidance  and  prompting  for  the  learner.  Early 

in  the  learning  activity,  it  is  important  for  the  learner 
to  receive  the  necessary  guidance  and  prompts  so  that  he 
may  continue  his  active  practice.  As  skill  increases, 
the  amount  of  guidance  and  prompting  reduces  to  the  level 
that  will  actually  be  encountered  in  the  job  environment. 

4.  Provide  feedback  to  the  learner.  Feedback  is  a funda- 
mental concept  in  the  ISD  process.  Feedback  is  best  when 
it  can  be  a natural  by-product  of  the  task  performance. 

In  this  way  the  feedback  can  be  virtually  continuous. 
Sometimes  it  is  necessary  to  provide  a learning  environ- 
ment that  is  relevant  but  artificial.  This  often  occurs 
In  specially  designed  training  devices  and  simulators. 

The  third  form  of  feedback  is  knowledge  of  results. 
Knowledge  of  results  often  occurs  at  the  end-point  in  a 
performance  and  may  take  the  form  of  hits  on  a target 

or  some  other  direct  record.  It  is  often  less  desirable, 
because  while  it  tells  the  results  of  the  learning 
effort,  it  does  not  provide  the  learner  with  Information 
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as  to  what  can  be  done  In  order  to  improve.  A fourth 
form  of  feedback  Is  simply  providing  the  student  with  a 
correct  answer  to  the  situation  or  problem  he  faces. 

He  Is  then  permitted  to  compare  his  answer  with  the 
correct  answer.  All  these  methods  of  providing  feed- 
back are  not  possible  in  all  learning  situations. 
Therefore,  if  the  learning  is  to  be  efficient  and 
effective,  care  must  be  taken  to  provide  the  specific 
type  of  feedback  needed. 

Earlier,  objectives  were  divided  into  four  categories:  mental 
skills,  information,  physical  skills,  and  attitudes.  In  this  block, 
each  of  these  learning  categories  is  further  divided  into  appropriate 
subcategories  so  that  more  specific  learning  events  and  activities  can 
be  specified.  There  is  a total  of  eleven  subcategories,  each  of 
which  has  a number  of  guidelines  for  the  design  of  instruction  for 
that  category  of  learning. 

Outputs 

The  outputs  from  Block  III.l  include  the  classification  of  the 
learning  objectives  into  their  appropriate  subcategories,  the  learning 
guidelines  chosen  for  groups  of  learning  objectives,  and  the  learning 
events  appropriate  to  the  learning  objectives. 

Management  Decisions 

Since  the  procedures  for  specifying  learning  events  and  activities 
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have  been  spelled  out  in  sufficient  detail  for  adequate  analyses  and 
classification,  the  manager  is  concerned  primarily  with  reviewing  the 
assignments  to  insure  that  the  objectives  have  been  correctly  classified. 


Incorrect  classifications  can  result  In  Inefficient  learning  and 
sometimes  result  In  learning  which  is  not  effective.  There  are 
relatively  few  trade-offs  to  be  negotiated,  and  resource  allocation 
problems  are  ordinarily  not  encountered  in  this  block. 


BLOCK  III. 2:  SPECIFY  INSTRUCTIONAL  MANAGEMENT  PLAN 
AND  DELIVERY  SYSTEM 


Introduction 

The  Initial  step  In  Block  III. 2 Is  the  specification  of  the 
Instructional  delivery  system.  Media  selection  is  a major  mear.s 
for  determining  how  the  Instruction  Is  to  be  presented  to  the  stu- 
dent. The  choice  of  the  media  mix  has  an  impact  on  both  the  effec- 
tiveness and  cost  of  training.  A systematic  approach  to  media 
selection  requires  consideration  of  the  nature  of  the  objectives,  the 
learning  category  of  each  group  of  objectives,  use  of  learning  guide- 
lines, and  use  of  learning  activities  based  on  those  guidelines. 

Because  instructional  requirements  are  different  for  different  learnin 
objectives,  equipment  and  facility  constraints,  cost  of  procurement, 
cost  of  replacement,  development  costs,  and  the  characteristics  of 
the  students  must  all  be  taken  into  account. 

Based  on  the  determinations  made  in  this  block,  existing  instruc- 
tional materials  will  be  examined  in  Block  III. 3 to  determine  whether 
they  are  suitable  for  instruction  in  a new  situation.  It  will  often 
be  necessary  to  make  some  adjustment  in  the  instructional  delivery 
system  in  order  to  accomodate  useful  high  quality  materials  discovered 
as  a result  of  this  search. 

A second  principle  step  in  this  block  is  the  preparation  of  the 
system  management  plan  which  is  the  principal  organizational  document 
for  the  instruction.  It  indicates  exactly  how  the  instruction  is  to 
be  conducted,  how  the  students  are  to  be  managed,  when  and  where  they 
will  be  tested,  what  the  instructors  and  other  support  personnel  are 

to  do,  and  how  each  of  the  many  elements  within  the  plan  work  together. 
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The  system  management  plan  Is  also  used  by  the  internal 
evaluator  to  plan  and  conduct  the  internal  evaluation.  It  is 
usually  necessary  to  develop  the  system  management  plan  with 
the  assistance  and  cooperation  of  the  internal  evaluator  to  be  sure 
what  Is  being  planned  can  be  properly  evaluated. 

Rationale 

While  the  media  and  delivery  systems  are  usually  selected  on  the 
basis  of  equipment  availability,  local  past  experience,  and  avail- 
able production  facilities,  there  are  other  important  considerations 
which  should  be  taken  into  account  to  make  better  long  range  plans. 
Techniques  have  been  developed  for  the  projection  of  procurement  and 
revision  costs  in  a wide  variety  of  instructional  media.  Frequently, 
there  can  be  a large  discrepancy  between  the  projected  costs  of  an 
existing  or  available  delivery  system  and  one  which  has  been 
optimally  designed. 

Large  discrepancies  of  this  nature  can  often  serve  as  the  basis 
for  management  decisions  to  embark  on  a new  approach  to  instruction 
when  it  can  be  shown  that  the  new  approach  will  have  important  pay- 
offs either  in  costs,  time  reductions,  or  increases  in  effectiveness. 
While  it  may  not  be  cost  effective  to  invest  in  a new  delivery  sys- 
tem for  a single  course  of  instruction,  such  an  investment  may 
have  a significant  pay-off  over  time.  Data  accumulated  through  time 
will  provide  an  important  source  of  information  for  management  in 
making  better  long  term  decisions. 

New  delivery  systems  and  techniques  often  become  fashionable  sim- 
ply because  they  are  available.  In  this  block,  procedures  are  defined 
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for  selecting  one  or  more  suitable  media  for  specific  learning  events 
and  activities.  By  using  this  approach,  delivery  systems  cam  be 
selected  on  the  basis  of  defined  requirements  rather  than  on  the 
basis  of  availability  or  the  appeal  of  currently  existing  fads. 

Inputs 

The  inputs  to  Block  III. 2 Include: 

1.  The  learning  events  and  activities  specified  as  a 
result  of  the  effort  in  Block  III.l. 

2.  The  setting  selection  criteria  and  rationale  de- 
veloped as  an  output  of  Block  1.5. 

3.  The  analysis  of  existing  course  information  developed 
in  Block  1.4. 

4.  Other  documentation  developed  in  prior  steps,  as  needed. 
Procedures 

The  first  step  is  to  identify  a list  of  media  mixes  or  altern- 
atives. A media  alternative  is  a form  of  instructional  material 
that  contains  the  stimulus  criteria  required  by  learning  activity. 

Each  of  these  correctly  identified  media  alternatives  becomes  a 
candidate  for  possible  use  as  part  of  the  final  delivery  system.  The 
learning  guidelines  may  require  a number  of  different  forms  of  presen- 
tation such  as  visual,  visual  motion,  high  quality  audio,  etc.  Those 
media  alternatives  which  can  meet  these  criteria  are  identified. 

In  the  ISO  manuals,  a number  of  matrices  have  been  developed  which 
assist  in  correctly  identifying  media  candidates  which  have  the 
characteristics  required  to  satisfy  the  learning  guidelines.  As  the 
learning  objectives  are  examined  and  classified,  there  may  be  an 
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indicated  need  for  a specific  medium,  or  there  may  be  an  Indication 
that  virtually  any  approach  might  be  satisfactory.  Both  kinds  of  in- 
formation are  important  because  some  forms  of  media  are  much  less 
expensive  than  others.  In  those  instances  where  two  or  more  media 
can  satisfy  the  learning  guidelines,  the  decisions  can  be  made  on 
the  basis  of  cost,  availability,  or  other  factors. 

Once  the  candidate  media  have  been  selected,  the  management  plan 
can  be  specified.  The  instructional  management  plan  can  be  usefully 
divided  into  two  general  categories:  those  employing  the  group 

block  scheduling  mode  and  those  using  self-paced  modes.  Group  block 
scheduled  instruction  has  existed  for  many  years  in  military  train- 
ing and  probably  is  the  most  widely  used. 

More  recently,  through  careful  application  of  systems  engineering 
or  ISD  techniques,  self-pacing  procedures  have  been  found  to  be  ex- 
tremely effective.  There  are  criteria  in  the  manual  which  assist  the 
instructional  designer  in  selecting  the  mode  of  instruction.  Because 
of  the  wide  differences  in  requirements  of  each  of  the  instructional 
settings,  the  management  plan,  must  be  carefully  put  together  to  meet 
the  requirements.  For  example,  while  group  management  plans  are 
usable  in  a resident  school,  or  perhaps  an  installation  support 
school  setting,  they  are  far  more  difficult  in  a formal  on-the-job 
training  program.  These  factors  are  taken  into  consideration  as  the 
management  plan  Is  developed. 

The  role  of  the  instructor  must  be  carefully  defined.  With  the 
increased  use  of  other  available  techniques  of  delivery,  it  is  no 
longer  an  absolute  necessity  to  use  an  instructor  solely  in  a talking 
or  demonstrating  mode.  More  recently,  the  job  of  the  instructor  has 
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Increased  In  complexity  to  the  point  where  he  is  more  often  thought 
of  as  a manager  of  instructional  resources.  This  may  mean  special 
preparation  or  training  and  possible  assignment  of  part  of  the  in- 
structional staff  to  provide  the  necessary  support  for  the  instructor 
in  the  manager  role. 

Finally,  necessary  facilities  and  instructional  resources  must 
be  identified  and  specified.  These  instructional  products  must  be 
classified  as  reusable  or  consumable;  that  is  some  of  them  may  be 
retained  by  the  student  for  his  future  use  and  others  may  be  used  up 
in  the  process  of  instruction. 

Outputs 

The  outputs  from  Block  III. 2 include: 

1.  The  system  master  plan  which  contains  all  of  the 
specifications  for  people,  resources,  facilities,  and 
other  requirements. 

2.  The  selected  media  and  the  media  alternatives  where  al- 
ternatives are  appropriate,  and  a summary  statement  of  the 
rationales  for  the  selections  that  have  been  made. 

Management  Decisions 

The  instructional  manager  will  be  heavily  involved  in  the  decision- 
making process  in  Block  III. 2 from  the  beginning.  Virtually  no  decision 
is  made  in  this  block  which  does  not  affect  a number  of  different  de- 
partments, commit  various  resources,  or  have  implications  for  the 
utilization  of  facilities  and  people.  The  manager  must  constantly  be 
aware  of  the  alternatives  being  considered  and  must  be  able  to  con- 
tribute to  the  decision-making  process  by  providing  information  about 
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facilities,  personnel,  and  other  resource  availability. 

Because  these  decisions  have  extremely  important  implications  for 
training  efficiency  and  effectiveness,  the  manager  must  cart.ully 
weigh  the  alternatives  within  the  context  of  the  currently  operating 
environment.  Another  important  management  function  with  regard  to 
Block  III. 2 is  the  accumulation  and  systematic  use  of  cost  data  for 
each  of  the  media  alternatives  considered.  While  there  are  many 
cost  models  available,  it  is  recommended  that  the  manager  become 
familiar  with  the  cost  accumulation  techniques  specified  in  the 
TECEP  volumes.  References  for  other  cost  techniques  are  included 
at  tiie  end  of  this  block. 

It  may  not  be  possible  to  make  completely  data-based  decisions 
with  respect  to  the  trade-offs  required  in  resources,  cost,  and 
effectiveness  of  the  alternatives  since  cost-effectiveness  infor- 
mation depends  on  valid  and  reliable  data  both  on  the  cost  of  in- 
struction and  the  outputs  of  instruction.  However,  it  is  possible 
to  make  generalizations  from  otiier  similar  situations  in  which 
people  have  researched  various  relevant  questions  about  the  cost  and 
relative  pay-offs  of  instructional  alternatives. 

Finally,  the  manager  will  have  a heavy  role  in  coordinating  the 
decision-making  processes  which  involve  many  other  departments . 
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BLOCK  III. 3:  REVIEW/SELECT  EXISTING  MATERIALS 


Introduction 

Reviewing  existing  learning  materials  for  the  purpose  of  de- 
termining their  potential  value  for  use  in  the  current  training 
program  is  accomplished  at  this  point.  A careful  search  is  made 
within  and  between  services  in  order  to  locate  as  many  candidate 
instructional  materials  as  possible. 

Having  made  a search  and  located  potentially  valuable  instruc- 
tional materials,  these  materials  are  reviewed  specifically  to  see 
whether  they  meet  the  learning  objectives  established  for  the  current 
training  and  whether  they  can  be  used  or  adapted  for  use  within  the 
context  of  the  selected  learning  guidelines  and  media. 

Candidate  materials  are  selected  or  adapted  for  use  where  appro- 
priate, and  are  rejected  when  they  fail  to  meet  the  current  needs. 

If  suitable  existing  materials  cannot  be  found,  new  materials  must 
be  developed  according  to  the  procedures  outlined  in  Block  III. 4. 

Rationale 

The  majority  of  training  programs  will  not  be  totally  new.  Most 
of  them  will  be  additions  to,  changes  to,  or  revisions  of  existing 
training  courses.  Even  if  the  potential  training  is  new  to  a given 
branch  or  service,  there  is  a good  chance  that  it  might  have  been 
offered  by  another  branch  of  the  service  before.  In  order  to  avoid 
duplication  of  effort  and  to  take  advantage  of  materials  developed 
elsewhere,  procedures  are  set  up  through  which  the  organization  which 
manages  each  DOS  can  accumulate  a catalog  and  library  of  existing  ma- 
terials appropriate  to  that  DOS. 
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Materials  which  do  not  meet  specification  exactly  may  often  be 
used  or  adapted  on  an  interim  basis,  while  more  suitable  instruction 
is  being  developed.  This  practice  will  allow  more  immediate  intro- 
duction of  the  training,  and  improvement  can  occur  as  newer  materials 
are  developed  when  time  and  resources  are  available. 

Inputs 

The  inputs  to  the  Review/Select  Existing  Materials  block  con- 
sist of: 

1.  Specification  for  learning  events  and  activities 
which  are  outputs  of  Block  III.l. 

2.  Specifications  for  the  instructional  management  plan 
and  delivery  system,  the  output  from  Block  III. 2. 

3.  Specification  of  student  entry  behaviors,  the  output 
from  Block  II . 3. 

4.  The  list  of  candidate  existing  courses  examined  by  the 
procedures  outlined  in  Block  1.4. 

At  this  point,  all  prior  documentation  will  ce  available  for  use, 
but  the  procedures  described  here  will  depend  most  heavily  on  the 
inputs  described  above. 

Procedures 

The  following  procedures  will  require  facilities,  people,  and  a 
reasonable  amount  of  time  for  an  adequate  search  and  review  to  be  made. 

In  order  for  an  exhaustive  search  to  be  made,  sources  of  infor- 
mation must  be  located  and  followed  up.  Local  commands  must  establish 
BOPs  which  will  enable  the  reviewer  to  communicate  with  other  commands 
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to  obtain  the  candidate  materials.  It  will  require  some  time  to 
locate  the  people  in  the  other  branches  or  other  services  who  have 
primary  responsibility  for  the  DOS  in  question.  Frequently,  brief 
telephone  discussions  can  greatly  reduce  the  amount  of  time  and 
effort  spent  in  reviewing  material  with  similar  titles  but  which  may 
serve  totally  different  purposes. 

Once  the  materials  have  been  collected  and  cataloged,  the  next 
step  is  to  review  the  materials  to  be  sure  they  match  the  learner 
characteristics,  the  learning  guidelines,  the  instructional  delivery 
system,  and  the  management  plan  for  the  training  being  developed. 

Since  one  of  the  more  important  considerations  in  the  selection  of 
media  and  management  plan  is  cost,  the  availability  of  existing  ma- 
terials can  often  have  an  important  impact  on  that  decision.  Acqui- 
sition costs  of  existing  materials  can  be  dramatically  lower  than 
development  costs  for  new  materials,  thus  providing  an  incentive  for 
consideration  of  existing  materials. 

Materials  can  be  selected  and  used  in  their  existing  form  or, 
if  they  have  important  contributions  to  make,  may  be  adapted  or  re- 
vised to  suit  current  needs.  The  reviewer,  with  the  advice  of  special 
ists  in  media  and  materials  can  often  make  a quick  decision  about 
the  feasibility  of  adapting  existing  materials  rather  than  using  them 
as  they  were  produced. 

A number  of  factors  must  be  considered  before  deciding  to  modify 
existing  materials.  First,  an  important  consideration  is  whether  the 
need  is  going  to  be  long  or  short  term.  It.  may  not  be  economical  to 
revise  existing  material  for  short  term  use.  The  physical  condition 
and  prior  history  of  the  materials  can  often  be  the  deciding  factor. 
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If  the  materials  are  not  in  good  shape  or  if  the  masters  are  not 
available,  revision  or  adaptation  may  be  more  expensive.  Ordinarily 
more  than  one  judge  should  be  used  for  any  piece  of  existing  ma- 
terials to  be  sure  that  all  of  the  relevant  factors  are  considered. 

This  will  almost  always  occur  when  the  reviewer  discovers  candidate 
material  and  submits  them  to  the  training  development  committee 
for  review. 

The  important  rule  to  follow  in  reviewing  existing  materials  is: 

Do  not  reject  materials  simply  because  they 
were  developed  elsewhere  and  do  not  accept  ma- 
terials simply  because  they  are  available. 

Outputs 

The  outputs  from  this  block  are  selected  materials  classified  into 
two  categories: 

1.  Existing  materials  that  are  adequate  for  use,  as  is,  and 

2.  Materials  which  must  be  revised  before  they  can  be  used. 

Management  Decisions 

In  order  to  obtain  long  term  benefits  from  this  block,  it  will  be 
necessary  for  management  to  establish  carefully  monitored,  local  SOPs 
that  can  be  readily  followed  in  obtaining  existing  materials  for  review. 
Further,  as  these  materials  are  accumulated  they  will  require  cata- 
loging and  storage  in  some  Form.  Either  the  actual  materials  can  be 
stored  or  a carefully  written  description  of  the  materials  can  be 
filed  and  cross-referenced. 

A second  form  of  management  decision  will  involve  making  the  neces- 
sary trade-offs  to  decide  whether  existing  materials  can  be  used  on  an 
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interim  or  long  term  basis  and  whether  it  is  worth  the  time  and  effort 
to  adapt  or  revise  them. 

Significant  factors  in  these  decisions  will  be  the  time  avail- 
able for  development,  tne  resources  available,  and  the  potential  for 
making  improvement  over  existing  materials  by  developing  new  ones. 

Virtually  every  consideration  in  the  management  decisions  in  this 
block  involve  costs.  Careful  applications  of  the  procedures  can 
have  very  favorable  cost  results  over  the  long  term. 


BLOCK  I II. 4:  DEVELOP  INSTRUCTION 

Introduction 

One  of  the  larger  efforts  in  the  ISO  process  is  that  of  developing 
instruction  to  accomplish  the  specified  learning  objectives.  At  this 
point,  all  available  off-the-shelf  instruction  has  been  selected  and 
included.  What  remains  is  to  complete  all  of  the  materials  necessary 
to  produce  the  desired  results  with  the  trainees.  From  Block  III. 2 
the  media  and  methods  of  Instruction  have  been  specified,  and  from 
Block  III. 3 the  available  materials  have  been  collected  and  assembled. 
This  is  the  appropriate  place  to  produce  all  the  additional  instruc- 
tional materials  which  will  be  used.  The  process  includes  develop- 
ing first  draft  materials  using  the  learning  events  specified  in  1 1 1 . 1 . 
These  first  draft  materials  are  tried  on  students  and  finally  sent  to 
the  appropriate  production  specialists  for  development.  A variety  of 
approaches  are  available  for  use.  The  appropriate  mix  of  these  ap- 
proaches will  depend  in  large  part  on  the  available  time  and  the 
facilities  and  resources  available  locally.  Lectures,  video  tapes, 
slide/tape  presentations,  job  performance  aids,  and  formal  on-the-job 
training  are  all  developed  according  to  the  same  general  principles. 
The  prescribed  procedures  allow  for  ample  internal  content  review  of 
the  materials  to  insure  that  they  are  doctrinally  correct. 


Fundamental  to  the  process  is  the  development  of  the  minimum 
instruction  necessary  to  accomplish  the  intended  learning  objectives. 

To  achieve  this  end,  a very  lean  approach  to  writing  initial  drafts 
is  required.  As  the  materials  are  tried  with  students,  weaknesses 
and  discrepancies  can  be  oentified,  and,  where  necessary,  materials 
can  be  expanded  to  overcome  any  shortcomings. 

Available  measurement  techniques  do  not  permit  the  easy  identi- 
fication of  instruction  which  has  more  material  than  is  required. 
Further,  the  elimination  of  already  prepared  materials  is  a more 
expensive  process  than  the  expansion  of  under-prepared  materials. 

Here,  tradeoffs  must  always  be  made  between  the  most  ideal  way  of 
approaching  the  Instruction  and  the  resources  available  to  accomplish 
these  ends. 

Inputs 

The  inputs  to  the  develonment  of  instruction  block  will  be  all  of 
the  documentation  available  from  the  processes  thus  far.  Any  or  all 
of  these  inputs  may  be  required  in  correctly  applying  the  procedures. 

Procedures 

Once  the  materials  have  been  organized  and  sorted  according  to  those 
materials  which  are  available  and  those  which  must  be  developed,  it 
is  necessary  to  separate  the  groups  of  learning  objectives  according 
to  the  selected  media.  In  some  instances,  more  than  one  form  of  in- 
struction will  have  to  be  developed  in  order  to  accomplish  the  learning 
objectives . 

The  initial  step  in  the  preparation  of  materials  involves  preparing 
the  first  draft  script  or  storyboard.  These  first  draft  scripts  and 
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storyboards  serve  the  purpose  of  Informing  production  staff  of  the 
requirements  for  production. 

For  some  materials  to  be  developed,  including  those  for  formal  on- 
the-job  training  and  for  job  performance  aids,  the  first  draft  script 
and  storyboard  may  be  replaced  with  a fully  descriptive  outline  of  the 
steps  to  be  followed. 

When  a small  amount  of  instruction  on  a learning  objective  has  been 
developed,  it  is  tried  with  a single  trainee  from  the  target  popu- 
lation to  see  whether  it  is  successful.  Since  these  materials  should 
have  been  prepared  in  the  leanest  possible  form,  the  tryouts  should 
reveal  weaknesses  and  areas  where  additional  instruction  is  required. 

Outputs 

The  outputs  from  Block  III. 4 include  all  of  the  materials,  pro- 
cedures, plans,  and  media  necessary  co  conduct  the  instruction. 

Since  the  output  of  Block  III. 4 goes  through  the  process  described 
in  Block  III. 5,  VALIDATE  INSTRUCTION,  the  final  effort  involved  in 
Block  III. 4 will  be  to  make  the  revision  based  on  the  data  collected 
in  Block  III  .5. 

Management  Decisions 

Probably  the  most  difficult  area  for  management  in  developing  in- 
struction will  be  to  coordinate  the  efforts  of  a variety  of  people 
in  a number  of  different  skills  areas.  Depending  upon  the  media  and 
method  ^elected,  the  manager  may  be  required  to  arrange  for  the  pro- 
duction of  video  tapes,  printed  materials,  audio  materials,  the  de- 
velopment of  training  aids  or  devices,  and  any  one  of  a number  of  other 
instructional  approaches. 
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An  almost  constant  process  of  trading  off  availability  of  facil- 
ities and  resources,  acceptable  levels  of  the  quality  of  the  instruc- 
tion, and  the  scheduling  of  completion  dates  will  occur  as  progress 
in  this  block  continues. 

In  many  instances,  it  will  be  necessary  for  the  manager  to 
challenge  decisions  made  earlier  on  the  basis  of  the  lack  of  facil- 
ities or  resources,  and  the  waiting  time  required  to  get  materials  pro- 
duced, and  in  combination  with  the  manager  of  the  effort  in  Block  III. 5, 
make  revision  decisions  or  redevelopment  decisions  following  test 
results. 

Further,  difficulties  for  the  manager  may  occur  when  trying  to 
get  existing  production  personnel  to  produce  materials  according  to 
the  needs  defined  by  ISO  processes  rather  than  according  to  the  tech- 
niques and  procedures  previously  used.  It  is  not  uncommon  for  media 
specialists  to  be  more  concerned  with  the  appearance  and  style  of  the 
production  than  with  its  instructional  effectiveness.  Thus,  the 
development  manager  must  constantly  be  aware  of  the  instructional  re- 
quirements of  the  materials  as  opposed  to  the  cosmetic  appeal.  Finally, 
as  new  materials  are  developed,  the  manager  must  insure  that  all  de- 
veloped materials  are  consistent  and  accurate  in  doctrine  and  content. 
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BLOCK  I II. 5:  VALIDATE  INSTRUCTION 

Introduction 

The  heart  of  the  development  phase  is  validating  the  instruction 
until  the  students  who  use  it  as  planned  meet  the  learning  objectives. 
The  validation  process  is  probably  the  most  powerful  procedure  in 
the  entire  developmental  effort. 

If  the  learning  materials  selected  in  Block  III. 3 and  those  de- 
veloped in  Block  III. 4 have  been  produced  efficiently,  they  will  have 
the  minimum  possible  elaboration.  The  instruction  should  be  "lean." 

When  this  material  is  tried  on  students  for  the  first  time,  it  should 
reveal  some  short-comings.  Students  ordinarily  wil 1 find  errors  in 
the  directions  or  will  fail  to  understand  or  be  able  to  meet  many 
of  the  requirements.  These  inadequacies  can  be  corrected  through  the 
process  of  revision.  If  on  the  other  hand,  tryout  reveals  only  a few 
errors  and  difficulties,  and  students  seem  to  grasp  everything  quickly, 
it  will  be  extremely  difficult  to  discover  if  the  instruction  has 
beer,  overdone. 

Selected  members  of  the  target  population  usually  go  through  the 
materials  individually  at  first  and  revisions  are  made  on  the  basis 
of  those  trials.  Following  the  initial  revision  cased  on  individual 
student  data,  the  number  of  students  is  increased  in  order  to  detect 
more  possible  errors.  Finally,  when  the  materials  are  thought  to  be 
complete,  they  are  tested  on  enough  students  so  that  their  effectiveness 
can  be  demonstrated  at  an  acceptable  level  of  confidence. 

As  the  materials  get  better,  fewer  and  fewer  students  will  have 
difficulties,  and  more  and  more  students  will  work  through  them  to  an 
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acceptable  level  of  performance.  At  this  point,  the  instruction  is 
ready  for  Introduction  into  the  setting  for  which  it  was  designed. 

Rationale 

Historically,  when  tests  have  been  administered  to  students,  the 
scores  have  been  entered  into  the  students  records  and  ultimately 
decisions  have  been  made  about  whether  the  student  should  be  passed 
or  retained.  In  some  instances,  these  tests  have  been  based  on 
training  requirements,  and  in  other  instances,  they  have  been  designed 
by  the  instructor  to  see  which  students  have  taken  the  time  to  learn 
the  more  irrelevant  and  obscure  points  in  the  course. 

In  the  ISD  process,  there  are  no  test  items  which  are  not  di- 
rectly related  to  objectives.  Because  of  this,  the  test  .results  can 
be  used  not  only  to  determine  if  the  student  has  passed,  but  also 
can  be  analyzed  to  see  which  of  the  instructional  areas  seem  to  be 
causing  the  most  problems.  Now,  the  same  data  that  were  originally 
used  only  to  evaluate  the  students  can  now  be  used  in  an  equally 
successful  way  to  evaluate  the  instruction. 

Analysis  of  this  data  points  clearly  to  those  areas  of  the  instruc 
tion  which  require  revision.  The  data  identify  the  need  for  revision 
but  only  a combination  of  those  familiar  with  the  content  of  the 
course  and  the  instructional  designers  will  be  in  a position  to  de- 
cide what  needs  to  be  done  as  a result  of  the  testing. 

This  validation  (or  formative  evaluation)  process  can  continue 
through  as  many  trainees  and  revisions  as  are  needed  until  the  in- 
struction is  serving  its  intended  purpose  to  an  acceptable  level. 

The  validation  process  is  suitable  for  all  known  forms  of  instruction. 


The  inputs  to  the  validate  instruction  block  include: 

1.  The  instructional  material  selected  in  Block  III. 3. 

2.  The  instruction  developed  in  Block  III. 4. 

3.  The  tests  developed  in  Block  II. 2. 

4.  Outputs  and  documentation  from  any  of  the  other  blocks 
required  for  analyses  of  the  materials. 

Procedures 

Initially,  it  will  be  necessary  to  make  a plan  for  conducting  the 
individual  trials.  Materials  will  have  to  be  collected,  the  tests 
must  be  assembled  and  ready,  and  the  trainees  must  be  available.  It 
is  common  to  collect  performance  data,  questionnaire  data,  and  in  some 
instances  interview  data  from  students  who  go  through  the  instruction. 

When  students  are  asked  to  complete  questionnaires  or  to  give  their 
opinions  about  the  materials,  it  is  important  to  be  sure  that  they 
are  asked  only  those  questions  to  which  they  have  the  capability  to 
respond.  Often,  the  students  can  describe  what  they  like  and  do  not 
like,  but  these  opinions  often  are  not  consistent  with  their  actual 
performance.  Students  often  cannot  make  recommendations  about  what 
needs  to  be  done  to  revise  the  materials,  but  they  can  describe  the 
point  at  which  they  had  difficulty  or  explain  what  it  was  that  confused 
them. 

Once  the  individual  trials  have  been  conducted,  the  data  are 
analyzed  by  the  instructi onal  developers  and  revision  decisions  are 
made  on  the  basis  of  the  tryout  information.  If  necessary,  additional 
individual  trials  are  conducted  until  the  individual  students  have 
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discovered  all  of  the  obvious  errors  and  difficulties.  Following  that, 
group  trials  are  conducted.  The  term  "group  trials"  does  not  neces- 
sarily imply  that  a large  number  of  individuals  must  all  be  present 
at  the  same  time.  It  simply  means  that  further  revisions  will  not 
be  made  until  a specified  number  of  individuals  have  been  given  the 
instruction  and  the  aggregrate  data  have  been  analyzed. 

In  the  manuals,  tables  are  provided  for  the  selection  of  the  num- 
ber of  students  required  in  order  to  determine  whether  the  instruction 
has  met  the  required  standards.  There  is  no  magic  number  which  can  be 
used  in  all  cases  since  confidence  in  the  results  is  a statistical 
rather  than,  an  absolute  determi nation. 

In  those  instances  where  the  objectives  are  extremely  important  or 
serve  as  the  basis  for  subsequent  important  instruction,  higher  per- 
formance standards  must  be  established  than  for  those  learning  ob- 
jectives which  are  independent  or  which  do  not  represent  critical  in- 
struction. Learning  objectives  must  be  classified  by  those  familiar 
with  the  content  so  that  reasonable  standards  can  be  adopted.  If  the 
standards  are  set  too  high,  vast  amounts  of  time  and  resources  car.  be 
wasted  trying  to  achieve  unrealistically  high  standards.  If  the  stan- 
dards are  set  too  low,  large  numbers  of  trainees  will  have  to  be  re- 
cycled because  the  materials  were  not  adequately  prepared. 

The  best  practice  has  been  to  consider  the  results  of  each  ob- 
jective based  on  the  relative  importance  of  that  objective  to  the 
instruction  as  a whole.  Different  confidence  levels  may  be  selected 
for  different  learning  objectives  in  a reasonable  and  systematic  way 
with  the  result  being  that  the  instruction  as  a whole  wll  meet  planned 
expectations . 
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After  the  materials  have  reached  their  required  standards  of  per- 
formance they  are  ready  to  be  reproduced  In  sufficient  numbers  to 
serve  the  trainee  population. 

Outputs 

The  principal  outputs  of  Block  III. 5 are  the  validated  materials 
themselves.  These  would  include  all  of  the  lectures,  videotapes, 
printed  materials  and  other  instruction  to  be  provided. 

A second  output  of  this  block  would  include  a description  of  the 
validation  process  followed  and  the  results  finally  obtained  with 
the  materials-.  The  second  output  is  of  principal  interest  to  the 
internal  evaluation  function  in  Block  V.l. 

Management  Decisions 

Block  III. 5 requires  moderate  to  heavy  management  inputs  and  de- 
cision making.  The  manager  must  be  concerned  with  ensuring  that 
instruction  has  been  developed  which  meets  minimum  requirements 
commensurate  with  available  resources. 

The  manager  will  need  to  decide  how  many  revisions  of  the  materials 
can  be  undertaken  and  still  meet  the  planned  course  schedule.  The 
manager  may  also  be  called  on  to  decide  whether  to  complete  the  vali- 
dation process  prior  to  the  time  that  actual  trainees  are  exposed  to 
the  materials  or  whether  to  use  actual  trainees  in  the  validation 
process.  In  part,  his  decision  will  be  based  on  the  availability  of 
the  necessary  members  of  the  target  population  and  the  amount  of  time 
available.  If  new  trainees  must  arrive  before  validation  processes 
can  begin,  revisions  to  instruction  may  have  to  occur  following  the 
initial  use  with  trainees.  While  this  is  not  the  most  desirable  course 
of  action,  the  same  end  result  can  be  achieved  through  continuing  revision. 


92 


An  extremely  Important  management  function  in  this  block  Is  that 
of  insuring  that  the  revisions  made  are  necessary  and  that  the  data 
have  been  carefully  analyzed  and  interpreted  before  these  revisions 
are  made.  Frequently  the  "easiest"  solution  is  simply  to  add  more 
materials  without  adequate  analyses  of  the  problem.  The  manager 
must  constantly  guard  against  this  tendency  to  add  material  when  it 
may  not  be  necessary, 

It  will  be  a continuing  duty  of  the  manager,  in  working  with 
those  who  develop  materials,  to  convince  them  of  the  necessity  of 
developing  barely  adequate  materials  in  first  draft  form.  The  reason 
for  this  requirement  is  that  it  is  extremely  difficult  to  Identify 
materials  which  have  been  over  developed,  but  it  is  simple  to  identify 
materials  which  are  inadequate.  Finally,  the  manager  must  insure 
that  the  trainees  used  in  the  validation  process  are  truly  repre- 
sentative of  those  for  whom  the  instruction  is  being  designed.  Revi- 
sions and  adjustments  in  the  instruction  are  made  on  the  basis  of 
data  collected  from  the  trainees  used.  If  these  trainees  are  incorrect- 
ly selected,  bad  revision  decisions  inevitably  result. 


BLOCK  IV. 1:  IMPLEMENT  INSTRUCTIONAL  MANAGEMENT  PLAN 


Introduction 

The  instructional  portion  of  the  system  management  plan  which 
was  developed  previously  according  to  the  procedures  outlined  in 
Block  III. 2 Is  now  ready  for  Implementation.  A complete  instruc- 
tional program  should  be  available  which  includes  clear  statements’ 
of  what  is  to  be  accomplished,  how  it  is  intended  to  be  accomplished, 
and  how  it  will  be  determined  whether  the  plan  was  accomplished. 
Refinements  and  additional  details  will  be  necessary  to  complete  the 
evaluation  plan. 

Any  additional  or  revised  requirements  that  have  come  about  due 
to  the  validation  procedures  in  Block  III. 5 may  be  made  to  the  in- 
structor's manual.  Here  it  is  necessary  to  select,  assign,  and  train 
instructors  in  any  of  the  techniques  or  procedures  with  which  they 
are  not  familiar.  Ordinarily,  only  instructors  who  are  successful 
graduates  of  regular  instructor  training  schools  would  be  selected. 

It  is  not  the  intent  of  this  block  to  give  details  of  how  to  train 
instructors. 

The  implementation  of  the  instructional  management  plan  occurs 
just  prior  to  the  time  that  the  first  students  are  actually  pro- 
cessed into  the  instruction  and  the  space,  equipment,  and  other 
resources  are  obtained.  Activities  involving  students  occur  in  Block 
IV. 2. 

Rationale 

While  it  appears  that  the  implementation  of  the  instructional 

management  plan  occurs  at  a fixed  point  in  a linear  sequence,  the  actual 
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planning  and  development  has  been  taking  place  for  some  time.  The  Im- 
plementation of  the  management  plan  is  the  terminal  step  in  planning 
and  preparation  just  before  regular  Instruction  begins.  Instructional 
management  plans  will  vary  considerably  from  those  involving  typical 
resident  school  instruction  to  those  involving  remote  formal  on-the- 
job  training  programs.  No  single  plan  will  serve  all  purposes. 

It  is  at  this  point  in  the  process  that  any  discrepancies  or  de- 
ficiencies in  what  has  gone  before  will  be  identified  and  corrected 
prior  to  the  time  that  the  students  begin  work  in  the  course. 

Inputs 

The  inputs  to  the  Implement  Instructional  Management  Plan  include: 

1.  The  instructional  management  plan  developed  in  Block 
III. 2. 

2.  All  of  the  materials,  procedures.,  tests  and  other 
necessary  components  of  the  total  instructional  program. 


Procedures 

The  first  thing  that  will  need  to  be  done  is  to  make  a complete 
inventory  and  checklist  to  be  sure  that  everything  necessary  for  the 
implementation  of  instruction  is  available.  If  necessary,  at  this 
point,  changes,  additions,  and  deletions  should  be  made  to  the  in- 
structors' manuals  and  students'  manuals  to  insure  that  they  are  com- 
pletely up  to  date. 

It  must  be  made  absolutely  clear  exactly  what  the  instructors 
are  supposed  to  do,  when  they  are  supposed  to  do  it,  and  any  specific 
or  unique  requirements  which  they  may  be  required  to  meet. 
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For  the  students'  manuals,  all  of  the  necessary  exercises,  in- 
struction, directions,  and  other  requirements  should  be  completed  or 
modified  if  necessary. 

The  next  step  is  to  identify  those  Instructors  who  are  going  to 
be  involved  in  the  implementation  effort  and  to  provid'  whatever 
additional  training  may  be  necessary  to  carry  out  the  instructional 
plan.  Most  often,  this  involves  rehearsal  and  review  of  the  uses  and 
functions  of  equipment  to  be  used,  practicing  demonstrations,  or 
rehearsing  test  administration  when  tests  have  unique  features. 

Finally,  the  procedures  become  administrative  in  nature:  time  and 

space  must  be  secured,  all  of  the  necessary  facilities  and  equip- 
ment must  be  secured,  everything  necessary  to  the  operation  of  the 
course  should  be  checked  to  be  sure  that  it  is  in  safe  working  con- 
dition. 

When  this  has  been  completed,  instruction  is  ready  to  begin. 
Outputs 

The  outputs  of  this  block  include  all  of  the  materials,  equip- 
ment, and  other  items  necessary  for  the  operation  of  the  course, 
staff  which  has  been  trained  to  operate  the  course,  and  specific  in- 
structions to  be  followed  by  each  person  in  the  implementation  effort. 

Management  Decisions 

Much  of  the  effort  required  by  management  in  this  block  will  be 
a part  of  efforts  begun  earlier.  There  is  often  a frenzy  of  activity 
which  occurs  just  prior  to  the  deadline.  The  manager  will  ordinarily 
find  himself  heavily  involved  in  a wide  variety  of  decisions  which 
must  be  made  and  executed  prior  to  the  time  that  students  arrive. 
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Planning  for  every  possible  contingency  might  require  more  time 
than  simply  managing  and  solving  the  problems  at  the  last  minute. 

Many  of  the  problems  to  be  solved  at  this  point  will  have  relatively 
straightforward  solutions. 

The  manager  should  expect  to  be  constantly  available  to  those 
who  need  him  during  those  weeks  and  days  just  prior  to  the  imple- 
mentation of  the  new  instruction.  Timely  decisions  at  this  point  can 
have  a dramatic  impact  on  meeting  the  deadline. 


BLOCK  IV. 2:  CONDUCT  INSTRUCTION 


Introduction 

This  is  the  first  of  four  continuing  steps  in  the  ISO  process. 
This  block  is  followed  by  internal  evaluation,  external  evaluation, 
and  revise  system.  These  four  functions  occur  continuously  so  long 
as  there  is  need  for  graduates  from  a course. 

Each  command  using  these  procedures  will  have  already  established 
instructor  training  programs  which  provide  the  necessary  skills  for 
usua1  instructional  activities.  In  addition,  in  Block  IV. 1 any 
special  instructor  training  would  have  been  conducted.  At  this 
point,  what  remains  to  be  done  is  to  conduct  the  instruction  accord- 
ing to  the  instructional  management  plan.  The  instruction  can  be 
conducted  in  resident  schools,  installation  support  schools,  formal 
on-the-job  training,  self-teaching  exportable  packages,  or  whatever 
was  specified  for  meeting  the  intended  objectives  of  the  course. 

As  the  instructor  is  provided  with  more  and  more  resources  and 
training  in  their  use,  his  role  increases  beyond  that  of  traditional 
classroom  presentations  to  include  that  of  an  instructional  manager. 
In  self-paced  and  other  nontraditional  courses,  the  role  of  the  in- 
structor will  be  even  more  vital  to  the  attainment  of  the  objectives. 

The  instructor  will  manage  resources,  make  presentations,  admin- 
ister tests  to  students,  record  data,  and  make  recommendations  for 
improvement  to  the  instruction. 

Working  in  cooperation  with  the  internal  evaluation  group,  the 
instructor  will  help  evaluate  the  students  and  the  instruction. 
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Rationale 

The  rationale  for  the  Conduct  Instruction  function  is  very 
straightforward.  The  instructor  in  direct  contact  with  the  students 
will  make  presentations,  give  demonstrations,  provide  the  students 
access  to  the  learning  resources,  administer  tests,  and  keep  what- 
ever rer  -is  are  necessary.  It  is  the  instructor's  .job  to  make  the 
instruct'  ark  and  to  keep  detailed  records  of  any  problems  that 
are  encountered. 

Inputs 

The  inputs  to  the  Conduct  Instruction  function  include  all  of 
the  instructional  materials  prepared  in  the  pricr  steps,  all  of  the 
required  facilities  and  resources,  instructors’  manuals,  and  all  of 
the  tests,  rating  scales,  questionnaires,  and  procedures  defined  in 
the  instructional  management  plan. 

Procedures 

The  procedures  for  conducting  instruction  will  depend  principally 
upon  the  selected  instructional  setting,  the  methods  and  media  chosen, 
and  the  management  plan  within  which  the  instruction  will  operate.. 

The  instructor  may  be  required  to  make  presentations,  give  lectures, 
give  demonstrations,  interview  students,  provide  tutorial  assistance, 
administer  performance  measures  and  other  forms  of  tests,  and  provide 
inputs  to  the  internal  evaluators  regarding  all  the  elements  of  in- 
struction with  which  he  has  contact. 
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Outputs 

The  outputs  of  the  Conduct  Instruction  function  include: 

1.  Students  who  have  graduated  or  completed  the  in- 
struction as  required. 

2.  Specific  ratings  or  evaluation  of  the  instructional 
material  or  procedures. 

3.  All  of  the  data  collected  as  a part  of  the  Conduct 
Instruction  function  including  student  test  results, 
student  ratings,  student  critical  incident  reports, 
student  time  records,  and  other  data  upon  which  the 
course  revision  and  operations  decisions  will  be  made. 

Management  Decisions 

Managing  the  instruction  is  a ^unction  in  which  there  is  already 
considerable  management  experience  and  expertise.  SOPs  and  other 
guidance  exist  in  all  commands  indicating  how  the  instructional  mana- 
ger should  proceed  vith  his  duties. 

In  addition  to  the  traditional  administrative  requirements  of 
making  sure  that  all  the  instructors  perform  as  expected,  the  manager 
of  instruction  in  self-paced,  peer-tutored,  computer-assisted,  and 
other  nontraditional  forms  of  instruction  will  often  find  his  rcle 
expanded  to  Include: 

1.  Counseling  and  reassuring  instructors  who  are  unfamiliar 
and  uncomfortable  with  newer  instructional  techniques. 

2.  Providing  encouragement  and  support  to  those  instructors 
who  continue  the  efforts  as  planned  until  sufficient 

data  can  be  collected  in  order  to  make  reasonable  revision 


decisions . 
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3.  Meeting  regularly  with  the  internal  evaluation  group 
to  coordinate  data  so  that  evaluations  can  be  com- 
pleted and  revision  decisions  can  be  made. 

4.  Insuring  that  there  are  sufficient  learning  materials 
and  resources  available,  and  that  there  is  adequate  time 
to  complete  the  instruction  as  planned. 

5.  Managing  the  personnel  and  physical  resources  with  con- 
stant attention  to  ways  in  which  resources  and  facili- 
ties can  be  saved  and  personnel  resources  can  be  freed 
from  more  routine  duties  to  make  their  time  available 
for  other  necessary  ISD  functions. 


PHASE  V: 
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BLOCK  V.l:  CONDUCT  INTERNAL  EVALUATION 


Introduction 

Internal  evaluation  Is  planned  and  conducted  primarily  to  de- 
termine whether  the  ISD  course  development  effort  has  been  accomplished 
Data  are  collected  not  only  to  assess  student  progress  but  also, 
more  importantly,  to  assist  in  improving  the  quality  of  instruction. 

The  principal  question  to  be  answered  in  internal  evaluation  is:  Is 

the  instruction  providing  the  students  with  the  necessary  knowledge 
and  skills  to  meet  the  objectives  in  a satisfactory  manner? 

In  addition,  the  other  aspects  of  instruction  are  carefully  ex- 
amined. To  what  extent  did  the  development  effort  conform  to  an  accept 
able  ISD  procedure?  How  much  time  was  required  for  the  student  to 
complete  the  instruction?  Was  the  prerequisite  information  and  entry 
test  data  used  appropriately?  L.-'d  the  instructiona'  personnel  and 
test  administrator  perform  in  a way  that  was  consistent  with  the 
management  plan,  instruction,  and  testing  procedures? 

It  is  the  duty  of  the  internal  evaluator  to  collect  as  much  use- 
ful information  about  the  operation  of  the  course  as  possible  in 
order  to  increase  the  chances  of  making  greater  improvement  through 
time.  The  evaluation  process  consists  of  collecting  pertinent  progress 
and  process  data,  performance  data,  and  information  from  students,  in- 
structors and  other  personnel,  evaluating  their  data,  and  making 
recommendations . 

Rationale 

Two  evaluation  functions  are  recognized  in  the  control  phase  of 

the  ISD  process.  The  first  of  these,  internal  evaluation,  must  provide 
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the  basis  for  deciding  whether  the  course  met  its  planned  objectives. 
In  addition,  time  data  and  other  information  from  students  are 
collected  to  provide  the  basis  for  making  revision  decisions. 

Evaluation  is  not  unique  to  the  ISD  process,  but  nc  set  of  in- 
structional design  and  development  procedures  can  be  called  ISD 
unless  it  includes  an  adequate  internal  evaluation. 

Ideally,  the  internal  evaluator  would  be  assigned  to  a depart- 
ment not  directly  responsible  to  the  instructional  design,  develop- 
ment, or  delivery  system  manager.  The  more  independent  the  evaluator 
can  be,  the  more  accurate  and  useful  his  data  will  tend  to  be.  An 
impartial  evaluator  will  be  in  the  best  position  to  verify  that 
correct  procedures  have  been  followed  in  obtaining  the  results  of 
instruction. 

It  is  the  evaluator's  purpose  to  remove  as  much  of  the  guessviork 
as  possible  from  school  operation.  By  careful  analysis  of  the  eval- 
uation data,  a determination  can  be  made  as  to  the  problems  with 
the  instruction,  and  appropriate  revisions  can  be  made. 

Inputs 

The  inputs  to  the  Conduct  Internal  Evaluation  block  include: 

1.  Output  data  from  each  of  the  blocks  in  the  model. 

2.  The  data  collected  on  the  performance  of  the  student 
including: 

a.  The  results  of  the  tests, 

b.  The  time  required  to  complete, 

c.  Student  evaluative  data,  and 
c,  Instructor  evaluative  data. 


106 


Procedures 

The  first  step  in  the  internal  evaluation  function  is  concerned 
with  the  development  of  the  internal  evaluation  plan  which  was  re- 
quired in  Block  IV. 2.'  ISO  procedures  cannot  be  totally  sequential. 

That  is,  having  an  evaluation  plan  prior  to  the  conducting  of  in- 
struction is  necessary  if  appropriate  student  and  instructor  data  is 
to  be  collected. 

A part,  of  the  internal  evaluation  plan  is  the  progress  evaluation 
plan.  The  progress  of  the  development  of  instruction  is  monitored 
to  be  sure  it  is  consistent  with  the  required  procedures  and 
whether  or  not  the  effort  is  on  schedule. 

A process  evaluation  plan  is  prepared  in  which  the  evaluators  will 
check  the  actual  procedures  and  outputs  of  each  of  the  blocks  to  be 
sure  the  processes  are  consistent. 

An  extensive  documentation  checklist  is  provided  so  that  the 
internal  evaluator  can  compare  the  required  outputs  with  the  actual 
outputs  of  each  of  cne  u series. 

Each  of  the  areas  of  student  evaluation  is  spelled  out  in  detail. 
Course  requirements  are  compared  with  student  entry  behaviors  and 
a number  of  evaluation  designs  are  considered  in  order  to  answer 
specific  questions  about  the  instruction.  The  evaluator  must  determine 
whether  the  students  are  being  given  instruction  on  topics  they  have 
already  mastered,  or  whether  they  are  being  given  instruction  for 
which  they  did  not  have  the  entry  skills. 

Extensive  student  questionnaire  and  instructor  questionnaire 
data  are  collected  to  pinpoint  the  good  and  bad  aspects  of  the  in- 
struction. 
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Procedures  are  established  for  determining  whether  students  sat 
isfy  the  terminal  requirements  fo  the  instruction,  and  evaluation 
designs  are  used  to  isolate  specific  problems  within  the  course. 

Outputs 

The  outputs  required  in  Block  V.l  Include  the  following: 

1.  Complete  summaries  of  student  records  containing 
pertinent  information  on  the  students  who  took  the 
course. 

2.  The  internal  evaluation  report  containing  at  least 
the  following  information: 

a.  The  degree  to  which  the  ISD  process  was 
followed,  including  documentation  of  any 
exception. 

b.  The  developmental  time  and  resources  required 
to  produce  the  instruction. 

c.  A completely  detailed  and  a summarized  pre- 
sentation of  student  performance  data  organized 
by  terminal  learning  objectives. 

d.  A summary  of  the  major  inputs  from  the  in- 
structors including  the  analyses  made  of  the 
critical  incident  reports  of  questionnaires. 

e.  A summary  of  the  complete  findings,  their  in- 
terpretations, and  specific  recommendations  for 
revisions  based  on  those  findings. 
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Management  Decisions 

Probably  the  most  difficult  task  for  the  manager  who  must  see  to 
it  that  the  internal  evaluation  is  done  properly  will  be  to  establish 
a good  working  relationship  with  other  managers  and  involved  personnel. 
There  is  often  a very  real  concern  among  personnel  whose  responsi- 
bility it  is  to  design,  develop,  and  deliver  instruction  that  the 
evaluators  are  primarily  concerned  with  placing  blame.  To  increase 
the  likelihood  that  his  job  will  be  successful,  the  manager  must 
establish  a good  working  relationship  with  other  departments  and  must 
be  convincing  in  offering  assistance  and  interpreting  evaluative  re- 
sults. It  is  not  the  purpose  of  internal  evaluation  reports  to  place 
blarne  on  any  department  or  any  individual. 

i 

The  evaluation  is  primarily  the  basis  upon  which  instructional 
accountability  is  based.  Thus,  the  manager  should  make  a diligent 
effort  to  ne  objective  in  the  collection  and  interpretation  of  re- 
sults and  should  base  his  decisions  and  recommendations  on  the  collected 
data.  The  job  is  primarily  one  of  determining  whether  the  instruction 
met  the  expectations-  Thus,  it  is  primarily  the  results  of  instruction 
with  which  he  must  be  concerned  rather  than  the  processes  of  instruc- 
tion. 

The  manager  of  the  internal  evaluation  effort  will  be  heavily  in- 
volved in  the  development  of  the  evaluation  plan,  particularly  in  the 
progress  evaluation  plan. since  this  information  will  often  be  very 
important  to  higher  levels  of  management,  lhere  will  be  many  trade- 
offs required  in  attempting  to  obtain  the  most  useful  evaluative 
information  within  the  available  resources. 
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* 1 Finally,  the  internal  evaluation  manager  will  be  required  to 

present  and  defend  his  conclusions  about  the  performance  of  the 
instruction.  This  will  ordinarily  take  place  in  a meeting  with 
1 • all  the  other  managers. 


BLOCK  V. 2:  CONDUCT  EXTERNAL  EVALUATION 


Introduction 

External  evaluation  is  conducted  In  order  to  determine  whether  the 
output  of  the  Instruction  in  the  form  of  trainees  who  have  been 
sent  to  the  field  is  serving  the  needs  of  the  using  command.  The 
external  evaluator  provides  the  fundamental  data  for  quality  control. 
Those  students  who  have  met  all  of  the  instructional  requirements 
are  followed  into  the  field  either  physically  or  by  questionnaire. 
Performance  is  then  determined  by  job  performance  measures,  and 
supervisor^  evaluations  are  taken  into  account. 

Ideally,  the  external  evaluator  would  base  his  conclusions  prin- 
cipally upon  actual  job  performance  as  measured  by  the  JPMs  produced 
in  Clock  1.3  of  the  ISO  model.  Often,  because  of  scheduling  pro- 
blems, DOS  testing,  and  other  similar  difficulties,  some  data  will 
have  to  be  collected  in  another  way. 

Finally,  the  external  evaluation  report  is  prepared.  It  contains 
the  conclusions  and  the  data  upon  which  the  conclusions  were  based. 
Recommendations  for  any  necessary  revision  are  included  in  this  report. 

Rationale 

External  evaluation  is  required  to  be  sure  the  graduates  or  com- 
pleters of  the  instruction  meet  the  expectations  of  the  using  command. 
These  expectations  should  have  been  initially  determined  in  Block  1.1, 
ANALYZE  JOB,  and  Block  1.3,  CONSTRUCT  JOB  PERFORMANCE  MEASURES.  If 
the  job  has  been  carefully  defined,  the  appropriate  tasks  selected 
for  instruction,  and  the  job  performance  measures  are  consistent 
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with  performance  recommendations,  the  job  of  the  external  evaluator 
will  be  greatly  simplified. 

While  It  Is  normally  easier  to  evaluate  the  products  of  a course 
in  a resident  or  installation  support  school,  evaluation  of  the  use 
of  job  performance  aids  is  also  relatively  straightforward.  The 
requirements  to  provide  external  evaluation  on  the  performance  of 
students  who  have  been  in  formal  on-the-job  training  or  have  used 
self-teaching  exportable  packages  becomes  more  difficult. 

The  external  evaluator  should  also  make  recommendations  for  im- 
provements in  the  instruction  which  will  have  important  payoffs 
to  the  using  command. 

Inputs 

The  inputs  to  the  Conduct  External  Evaluation  block  include: 

1.  The  internal  evaluation  report  (or  the  available 
internal  evaluation  data). 

2.  The  output,  of  Phase  I including  all  of  the  job 
analyses,  task  selection,  job  performance  measures, 
and  setting  selection  information. 

3.  Any  documentation  from  management  or  higher  head- 
quarters which  has  the  effect  of  changing  the  require- 
ments of  the  course. 

Procedures 

The  initial  step  in  conducting  an  external  evaluation  is  the 
development  of  an  external  evaluation  plan.  The  plan  will  include 
the  specifications  for  the  activities  that  will  be  undertaken  in 
order  to  obtain  data  from  trainees  who  have  received  the  instruction, 
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and  from  appropriate  supervisors  and  managers.  To  the  fullest  ex- 
tent possible,  the  plan  will  call  for  the  collection  of  real  per- 
formance data  collected  as  a result  of  the  administration  of  job 
performance  measures,  and  other  indicators  of  performance  which 
might  include  such  things  as  advancement. 

Either  by  interlvew  or  by  questionnaire,  trainees  who  have 
completed  the  program  will  be  asked: 

1.  How  well  they  believe  they  are  able  to  perform  the  job 

2.  The  kind  and  amount  of  training  received  since 
arriving  on  the  job 

3.  How  well  the  instruction  prepared  them  for  the  job 

4.  The  portions  of  the  instruction  which  were  relevant 
to  the  job 

5.  Which  tasks  seem  to  cause  the  most  difficulty 

From  the  supervisors  of  graduates,  information  will  be  collected 
such  as: 

1.  How  well  the  graduates  are  performing  on  the  job 

2.  How  these  graduates  compared  to  those  who  received 
another  form  of  training 

3.  In  which  areas  the  graduates  were  inadequately  prepared 

From  other  sources,  including  an  evaluation  team,  the  following 

information  will  be  collected: 

1.  How  well  graduates  scored  on  the  job  performance  measures 

2.  Which  JPMs  gave  them  the  most  trouble 

3.  How  well  the  JPMs  were  administered 

4.  How  well  the  supervisor  knows  his  job 
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Ordinarily,  It  would  not  be  useful  to  contact  someone  who  has 
received  training  until  after  he  has  been  on  the  job  for  a period  of 
thirty  (30)  days  or  more.  This  may  not  be  a necessary  criterion  when 
formal  on-the-job  training  is  the  means  of  instruction. 

If  questionnaires  are  to  be  used,  they  must  be  developed  and 
tried  out  on  a limited  number  of  individuals  prior  to  the  time  they 
are  sent  to  all  of  those  persons  who  received  the  training.  This 
initial  tryout  will  eliminate  some  of  the  problems  and  confusion  with 
the  questionnaire.  Questionnaire  data  is  far  more  dependable  if  a 
large  percentage  of  the  questionnaires  mailed  out  are  returned.  A 
method  for  obtaining  a high  percentate  of  questionnaire  returns 
should  be  worked  out  as  a part  of  the  external  evaluation  plan. 

If  personal  interviews  are  to  be  used,  those  who  are  going  to 
do  the  interviewing  will  be  required  to  spend  some  time  going  over 
the  Interview  form  and  in  necessary  training  and  orientation  of  pro- 
per techniques  of  interviewing. 

Once  the  information  has  been  collected  about  the  performance  of 
trainees  on  the  job,  it  will  be  necessary  to  analyze  the  data  to 
draw  conclusions,  and  to  make  recommendations  based  on  those  conclu- 
sions. These  recommendations  will  ordinarily  be  in  the  form  of 
suggestions  for  revisions  or  changes  in  the  instruction,  although  many 
of  them  will  be  concerned  with  aspects  of  job  analyses  and  selecting 
tasks  for  training. 

Outputs 

The  output  from  the  Conduct  External  Evaluation  block  is  the  ex- 
ternal evaluation  report  which  contains  the  summaries  and  interpretations 
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of  the  collected  data,  the  methods  and  techniques  used  to  collect  the 
data,  the  problems  encountered  and  other  Information  which  will  be 
useful  to  management  in  making  the  decisions  about  revisions  to  the 
Instruction. 

Management  Decisions 

Management  decisions  in  the  external  evaluation  function  are  of 
two  general  classes:  short  term  and  long  term.  In  the  short  term, 

the  manager  will  be  principally  concerned  with  reviewing  the  altern- 
ative data  collection  procedures  and  trading  off  the  evaluation  needs 
with  the  resources  available.  One  of  the  more  important  assignments 
for  the  manager  of  external  evaluation  is  to  be  sure  that  the  quality 
of  the  data  collected  is  as  high  as  it  can  be.  If  course  revision 
decisions  are  going  to  be  made  on  the  basis  of  the  data,  it  must 
be  the  best  data  that  can  be  obtained. 

The  manager  may  wish  to  consult  evaluation  specialists  in 
order  to  be  sure  that  available  state  of  the  art  techniques  are 
employed  to  collect  this  information.  This  would  be  particularly 
true  in  the  construction  of  questionnaires  and  other  evaluative  in- 
struments as  well  as  the  evaluation  design  to  be  employed. 

In  long  term  evaluation, the  manager  will  be  concerned  with 
increasing  the  capability  to  collect  and  use  external  evaluation 
data.  In  some  commands,  the  return  rate  of  questionnaires  is  very 
low.  Decisions  based  on  low  return  rates  of  questionnaires  ordi- 
narily cannot  be  as  good  as  those  based  on  higher  return  rates. 

The  manager  also  may  be  concerned  with  attempting  to  develop  an 
external  evaluation  plan  to  review  the  performance  of  all  individuals 
in  a DOS  in  order  to  segregate  the  relative  contributions  of  the 
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various  forms  of  training.  It  is  the  external  evaluation  data  which 
ultimately  will  be  able  to  serve  as  the  basis  for  cost-effectiveness 
studies  when  comparing  training  settings.  In  order  to  achieve  this 
end  the  manager  may  wish  to  request  assistance  from  researchers  and 
evaluation  specialists  in  establishing  an  optimum  evaluation  system. 
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BLOCK  V. 3:  REVISE  SYSTEM 


Introduction 

At  the  completion  of  the  preparation  of  the  internal  and  external 
evaluation  reports,  the  data  contained  in  those  reports  must  be 
used  as  the  basis  for  deciding  which  elements  of  the  system  re- 
quire revision.  The  internal  and  external  evaluation  reports  will 
document  course  performance  on  an  internal  basis  and  the  external 
evaluation  report  will  have  documented  the  results  of  graduates  on 
the  job. 

A careful  analysis  must  be  made  of  the  data  contained  in  the 
reports  in  order  to  determine  the  need  for  revision.  Revision  needs 
occur  as  a result  of  changes  in  doctrine,  procedures,  weapons 
systems,  or  other  majcr  external  change  as  well  as  the  results  of 
the  performance  of  the  instructional  program.  Revision  can  occur 
in  any  part  of  the  system.  Revisions  can  be  undertaken  to  improve  per- 
formance of  the  students,  to  reduce  student  time  required  to  complete 
the  instruction,  or  to  try  to  retain  the  appropriate  level  of  effec- 
tiveness at  a lower  cost.  Estimates  of  the  potential  benefits  of 
revision  are  made  on  the  basis  of  the  evaluation  data.  The  specialists 
within  each  of  the  departments  concerned  with  revision  decide  upon  the 
tradeoffs  which  need  to  be  made  in  order  to  make  the  revision  worth- 
while. 

Rationale 

Data  based  system  revision  undertaken  as  a result  of  careful  con- 
sideration of  the  alternatives  is  the  heart  of  the  ISD  process.  The 
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ability  to  make  good  data  based  decisions  Is  dependent  upon  the  quality 
of  the  data  collected  and  the  care  with  which  appropriate  conclusions 
are  drawn  from  that  data.  It  is  ordinarily  through  the  revision 
process,  particularly  the  early  revision  cycles,  that  some  of  the 
great  payoffs  from  the  ISD  process  can  be  realized. 

An  extremely  careful  effort  must  be  undertaken  to  establish  the 
basis  for  comparing  instruction  after  revision  with  prior  operating 
data.  Since  comparative  data  are  ordinarily  stated  In  terms  of 
percentages  of  increase  or  decrease  in  time  efficiency,  percentages 
of  increase  or  decrease  in  effectiveness,  or  increases  or  decreases 
in  dollars,  these  numbers  must  be  adequately  documented  and  compared 
in  order  to  draw  appropriate  conclusions  from  the  results. 


Inputs 

The  inputs  to  Block  V.3  include  the  following: 

1.  The  internal  evaluation  report 

2.  The  external  evaluation  report 

3.  All  prior  course  documentation  needed 

4.  Necessary  external  inputs  which  may  arise  as  a 
result  of  changes  in  the  personnel  system  or 
other  sources  external  to  the  instructional 
setting. 

Procedures 

The  procedures  to  be  followed  include  the  following: 

1.  Determine  the  need  for  revision  based  on: 


a.  Changes  in  doctrine,  content,  new  system  require- 
ments, personnel  changes 
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b.  The  results  of  the  internal  evaluation  report 
indicating  some  deficiency 

c.  Results  of  the  external  evaluation  report  indi- 
cating graduates  are  not  able  to  do  the  job 

d.  Changes  In  training  requirements  or  resources 
which  require  changes  in  the  instruction 

2.  Determine  what  needs  revision.  It  is  absolutely  essential 
that  a careful  analysis  of  the  problem  discovered  in 

the  internal  or  external  report  be  made  prior  to  the 
time  revision  is  undertaken.  It  is  possible,  based  on 
uncertain  or  preliminary  data,  to  make  costly  and  unneces- 
sary revisions  to  perfectly  good  instruction. 

3.  It  is  important  to  make  a careful  analysis  and  projection 
tc  determine  whether  the  investment  required  to  decrease 
the  time  or  increase  the  effectiveness  of  instruction  is 
really  worthwhile  in  terms  of  the  potential  benefits  to 
be  obtained.  Any  resources  invested  in  making  changes  to 
existing  instruction  necessarily  means  that  other  aspects 
of  the  program  must  have  their  resources  reduced  or  de- 
nied. It  is  very  important  that  a clear  justification  ex- 
ists for  revision  before  it  is  undertaken. 

4.  Preparation  of  the  revision  plan.  The  revision  plan 
should  be  put  together  as  a result  of  conferences  that 
have  been  conducted  with  the  affected  departments.  If 
revisions  are  to  be  made  in  the  media,  the  course  manage- 
ment plan,  the  job  analysis,  or  whatever  section  of  the 
system  that  requires  attention,  those  who  must  be  charged 


with  the  accomplishment  of  the  revision  need  to  have 
the  opportunity  to  discuss  the  problems  and  potential 
benefits,  cost,  and  time  Involved  in  making  the  revision. 

5.  Once  the  revisions  have  been  made,  it  is  essential  that 
a follow-up  activity  ensure  that  the  expected  benefits 
from  the  revisions  have  actually  occured. 


Outputs 

The  outputs  from  the  Revise  System  block  include  the  following: 

1.  A complete  revision  plan  outlining  the  requirements 
for  revision  agreed  upon  among  the  affected  members  of 
the  staff 

2.  A revision  progress  plan  in  which  the  planned  revisions 
are  compared  with  the  actual  revisions  in  order  to  be  sure 
that  they  are  being  undertaken  and  completed. 

Management  Decisions 

Block  V . 3 represents  one  of  the  more  significant  areas  in  the 
IPISD  for  management  decision.  All  or  part  of  the  instruction  may 
require  revision.  Job  data,  task  selection  procedures,  JPMs,  and 
settings  may  all  require  serious  review. 

The  manager  must  decide  what  to  revise  in  the  context  of  con- 
sidering total  resource  allocation  and  scheduling  problems.  Within 
any  pool  of  resources,  allocating  a portion  of  them  to  revision 
means  that  amount  cannot  be  used  for  new  course  development.  Often, 
the  data  is  very  clear  and  the  decisions  can  be  made  with  confidence. 
Equally  often  there  will  be  inconsistencies  in  the  data  and  the  inferences 
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will  not  have  the  same  level  of  precision.  The  manager  must  con- 
tinue analysis  until  conclusions  can  be  reached  and  a course  of 
action  agreed  upon. 


LOCAL  INITIATIVES 


There  are  two  predictions  about  the  local  school  reactions  to  the 
IPISD  program  which  can  be  made  with  some  degree  of  safety.  First, 
that  the  IPISD  procedures  are  either  a little  or  a lot  different  than 
what  Is  being  done  at  the  school  now,  and  second,  that  there  will  be 
a number  of  specific  reservations  associated  with  the  implementation 
of  IPISD  at  the  school . 

Past  experience  has  indicated  that  schools  are  often  required 
to  follow  the  letter  rather  than  the  intent  of  the  procedures,  par- 
ticularly when  these  have  been  promulgated  by  regulation.  It  is  the 
intent  of  the  IPISD  implementation  plan  that  the  spirit  of  the  pro- 
cedure is  far  more  important  than  the  letter.  This  is  a troublesome 
notion  because  the  difference  between  letter  and  spirit  often  can  be 
detected  only  by  those  who  are  thoroughly  familiar  with  the  procedures. 
There  are  approaches  which  appear  on  the  surface  to  be  approximately 
equal  and  which  have  been  shown  to  be  quite  different. 

A second  reservation  is  that  of  becoming  involved  in  excessive 
paperwork,  forms,  and  reporting  requirements  to  the  detriment  of  the 
course  development  effort.  It  has  been  a specific  objective  in  the 
design  of  the  procedures  in  these  manuals  to  avoid  specifying  non- 
essential  forms  and  reporting  requirements-  Experience  with  the  pro- 
gram in  the  services  is  more  likely  to  result  in  a realistic  determi- 
nation of  essential  reporting  requirements. 

A third  reservation  is  that  suspense  dates  for  implementation  of 
new  programs  are  often  too  close  for  reasonable  compliance.  It  is 
expected  that  IPISD  will  take  considerable  time  and  effort  for  proper 
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implementation.  Experience  with  it  will  probably  provide  the  best 
indicators  of  when  the  procedures  should  become  completely  institu- 
tionalized. Further,  because  the  IPTSD  approach  Is  somewhat  different 
than  the  traditional,  it  will  take  time  and  resources  to  train  enough 
people  to  perform  the  work  in  accordance  with  the  1PISD  specifications. 
Time  must  be  provided  for  adequate  training. 

Because  the  procedures  in  the  Model  are  different  from  many  current 
practices  in  most  schools,  some  resistance  can  reasonably  be  expected. 
Common  questions  like,  "what's  wrong  with  what  we're  doing  now,"  and 
"we  tried  that  and  it  didn't  work,"  will  be  encountered.  Experience 
has  shown  that  as  people  become  more  familiar  with  the  ISD  approach 
and  use  it  as  a procedure  to  accomplish  specific  goals,  the  resis- 
tance to  its  use  declines.  Often,  sufficient  training  changes 
opponents  to  supporters.  Probably  one  of  the  more  general  fears  will 
be  the  lack  of  adequate  staff  to  continue  the  existing  work  and  to 
take  on  the  additional  tasks  of  implementing  IPISD  at  the  same  time; 
however,  implementation  of  IPISD  is  not  an  additional  task  which  will 
be  separately  funded.  It  is  an  evolutionary  process  which  will  ulti- 
mately replace  current  practices.  The  difference  between  what  is  being 
done  now  and  the  IPISD  approach  will  have  to  be  overcome  through  time 
as  resources  are  reallocated  to  accomplish  the  change.  The  implemen- 
tation is  planned  to  be  accomplished  through  the  reallocation  of 
existing  resources  rather  than  the  addition  of  new  resources. 

Spirit  of  the  Model 

It  is  a distinctly  stated  objective  of  the  interservice  implementation 
program  to  try  to  avoid  making  the  IPISD  program  a paper  exercise.  The 
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procedures  and  techniques  are  too  valuable  to  be  Ignored  or  rendered 
ineffective  by  absolute  compliance  to  rules. 

There  are  several  principles  and  concepts. which  underlie  the 
IP  ISO  Model.  It  Is  these  concepts,  principles,  and  rules  which  de- 
fine the  basic  intent  of  the  Model.  These  are  as  follows: 

1.  The  complete  and  full  use  of  the  management  information 
system  must  be  taken  in  order  to  arrive  at  clear  data- 
based  decisions  about  Instruction.  These  decisions  should 
range  from  defining  what  it  is  that  is  to  be  taught,  to 
how  it  is  to  be  taught  and  how  it  should  be  evaluated. 

2.  A sound  rationale  for  these  decisions  is  based  on  estab- 
lished principles  of  ISO.  If  current  practices  are  not 
defensible  in  the  context  of  an  ISD  approach,  they  should 
be  seriously  brought  into  question. 

3.  Basic  to  any  systems  approach  model  is  the  generation 
and  consideration  of  alternative  approaches  and  solutions 
to  a defined  problem.  These  alternatives  should  include 
instructional  strategies,  testing  procedures,  delivery 
systems,  and  student  management  systems.  The  selection 
of  the  optimum  alternative  is  one  principal  goal  of  the 
systems  approach. 

4.  Optimizing  course  effectiveness  and  time  are  basic  goals 
of  IPISD,  and  instructional  decisions  need  to  be  made 

on  the  basis  of  sound  cost  estimates. 

5.  Fundamental  to  the  IPISD  process  is  a revision  procedure 
which  requires  that  any  course  be  revised  based  on  usage 
data  collected  Internally  or  on  follow-up  data,  particularly 
if  the  course  has  failed  to  meet  its  stated  objectives. 
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If  these  basic  principles  and  practices  are  followed,  good  imple- 
mentation of  the  spirit  of  ISO  will  occur.  Following  the  specified 
procedures  will  not  always  yield  a good  solution  to  a problem  with- 
out that  element  of  judgment  which  is  the  final  determiner  of  what 
is  feasible  or  not  feasible. 

Realities  of  Implementation 

The  IPISD  program  does  not  provide  specific  procedures  for  every 
instructional  situation  that  can  be  encountered.  Some  situations 
are  covered  only  by  the  general  principles  underlying  the  Model.  It 
is  expected  that  local  applications  of  procedures  will  extend  the  Model, 
to  cover  local  requirements. 

Finally  one  of  the  more  difficult  problems  of  all  is  that  IPISD 
will  tend  to  reveal  shortcomings  of  current  practices  in  local  situa- 
tions. When  these  shortcomings  appear,  many  people  will  resist  change 
and  will  argue  against  the  Implementation  of  alternative  procedures. 

This  kind  of  reaction  must  be  taken  into  account  and  planned  for  in 
advance.  Remember  that  the  current  system  once  was  new,  but  eventually 
became  established  and  institutionalized.  In  time,  the  IPISD  pro- 
cedures also  will  become  institutionalized. 

Reducing  Costs 

Within  the  model,  there  are  procedures  which  tend  to  have  much 
greater  implications  for  the  trade-offs  of  effectiveness,  time, 
efficiency  and  cost  than  others.  A prime  example  of  a high  payoff  area 
is  the  process  of  selecting  tasks  for  training.  Any  task  selected 
for  training  will  continue  to  accumulate  costs  for  a long  period  of 
time.  Therefore,  it  is  absolutely  essential  that  a rigorous  procedure 
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be  established  for  selection  of  tasks  for  training.  Any  task  that  can 
reasonably  be  eliminated  has  the  potential  for  allowing  great  savings. 

Another  example  of  a high  payoff  area  lies  in  the  Instructional 
management  plan  and  delivery  system.  Careful  matching  of  the  manage- 
ment plan  and  delivery  system  can  lower  the  internal  costs  for  achieving 
certain  objectives.  The  relative  payoff  of  each  of  the  steps  in  the 
Model  is  described  in  more  detail  in  the  section  dealing  specifically 
with  that  step.  Remember,  controlling  unnecessary  costs  ts  one  of  the 
more  certain  ways  to  promote  real  economies. 

Management  of  ISP 

The  implementation  and  use  of  IPISD  procedures  will  probably  re- 
quire changes  in  existing  management  planning  and  decision-making 
procedures.  These  changes  will  be  particularly  noticed  at  the  local 
school  level.  Using  ISD  has  effects  on  both  the  resource  allocation 
and  supervisory  functions  of  management. 

As  with  most-other  functions,  long  term  planning  in  ISD  can  be 
expected  to  result  In  considerable  improvements  in  payoffs.  Because  many 
of  the  functions  called  for  in  IPISD  represent  professions  in  and  of 
themselves,  a strong  commitment  to  organizational  development  will 
be  required  to  meet  the  long  term  needs.  For  example,  job  analysis, 
measurement,  selection  of  media,  and  cost  effectiveness  analysis  are 
al  1 extremely  difficult  and  complex  fields.  Developing  a local  capa- 
bility to  work  in  these  areas  can  have  a very  high  pay-off  for  local 
commands. 

Because  of  the  specific  nature  of  many  of  the  functions  of  ISD,  the 
manuals  have  been  functionally  organized  to  provide  the  maximum 
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flexibility  at  the  local  level.  While  a series  of  steps  are  called 
for  In  the  development  of  any  Instruction,  the  1PISD  Model  does  not 
specify  what  kind  of  organization  is  necessary  to  perform  those  steps. 
Further,  the  Model  often  provides  alternative  approaches  to  the  same 
goals.  By  carefully  articulating  the  strengths  and  weaknesses  of  the 
local  school,  the  manager  can  improve  the  total  performance. 

Finally,  by  taking  Into  account  all  the  specific  management 
decisions  previously  described  in  the  Block  Summaries  it  is  possible 
to  continue  the  improvement  process  through  time. 

Where  to  Begin 

Any  school  manager  who  has  as  his  responsibility  the  administration 
of  a large  number  of  courses  is  faced  with  a difficult  problem  of 
selecting  and  establishing  priorities  for  courses  to  put  into  IPISD. 
Obviously  some  courses  will  not  be  discretionary,  they  will  simply  be 
required  by  a higher  command.  This  section  deals  with  the  use  of  dis- 
cretionary resources  in  local  commands. 

One  criterion  for  the  selection  of  courses  would  be  to  estimate  the 
potential  payoff  to  the  school  if  the  course  could  be  improved.  Here, 
payoff  refers  to  the  establishment  of  an  increasing  pool  of  discre- 
tionary resources.  If  it  were  possible  to  teach  a basic  course  with 
fewer  people  after  IPISD  than  was  required  before,  the  reassignment  of 
those  people  to  other  important  tasks  would  be  possible.  One  payoff 
then  is  to  organize  the  I P ISO  effort  so  that  the  maximum  number  of 
discretionary  resources  are  realized. 

One  of  the  more  obvious  places  to  look  for  these  high  payoff  courses 
is  in  the  area  of  "high  flow  - high  level"  courses.  These  are  courses 
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for  officers,  senior  enlisted  technical  course,  etc.,  where  the  Indi- 
viduals likely  to  be  In  the  course  have  good  academic  skills.  At  the 
opposite  end  of  the  continuum  would  be  a low  flow  course  for  trainees 
with  limited  academic  abilities. 

Another  consideration  might  be  to  select  a course  which  has  a 
high  flow  but  which  for  one  reason  or  another  has  not  produced 
graduates  who  are  satisfactory  to  the  receiving  commands.  In  this  case, 
the  objective  would  be  to  improve  the  performance  of  the  graduates, 
prior  to  attempting  to  realize  cost  or  time  reductions. 

The  decision  to  proceed  with  ISO  development  of  a course  must  be 
carefully  reviewed.  The  expected  outcomes  must  be  specified  and  sub- 
jected to  a critical  analysis.  First,  is  it  reasonable  to  believe 
expected  accomplishments  can  actually  be  achieved?  Second,  if  the 
objectives  are  achieved,  will  it  really  make  a difference  in  the 
school  or  recieving  command: 

1.  Will  it  reduce  time? 

2.  Will  it  be  less  costly? 

3.  Will  performance  meet  field  expectations? 

4.  Will  it  use  fewer  resources? 

In  planning  IPISD  courses,  careful  problem  analyses  must  be  made 
to  be  sure  that  the  course  is  planned  and  designed  to  meet  the  expecta- 
tions for  it.  In  order  to  realize  cost  savings,  it  is  necessary  to 
specify  in  advance  as  one  of  the  planning  constraints  of  the  course  that 
costs  should  be  reduced.  Unplanned  cost  savings  are  rarely  realized. 

If  it  is  the  purpose  of  the  IPISD  effort  to  reduce  the  time  in  the 
course,  again  this  must  be  specified  in  advance  so  that  the  instructional 
developers  and  designers  can  know  ahead  of  time  that  the  intended  outcome 


Fundamental  to  training  setting  decisions  and  also  to  media  de- 
cisions is  reliable  and  accurate  cost  information.  Because  there  are 
so  many  variables  which  affect  the  coif,  of  training  in  any  setting  or 
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media,  it  will  be  necessary  to  develop  appropriate  cost  information 
as  these  costs  can  be  established  and  verified. 

The  Navy's  Training  Analysis  Evaluation  Group  has  been  working  for 
a number  of  years  on  appropriate  media  selection  and  training  setting 
costs.  The  basic  document  summarizing  these  procedures  entitled, 

A Technique  for  Choosing  Cost.  Effective  Delivery  Systems  (TECEP) 
is  available  for  use.  This  document  can  be  used  by  local  managers 
either  in  its  current  form  or  as  adapted  to  local  requirements  as  a 
basis  for  generating  appropriate  training  cost  information.  This 
data  will  be  needed  in  several  of  the  steps  in  the  Model,  particu- 
larly in  Blocks  1.5,  III.l,  and  III. 2.  In  those  blocks,  specific 
reference  is  made  to  the  need  for  obtaining  and  comparing  alternative 
costs  for  different  training  approaches.  The  recommended  procedure 
is  that  the  technician  who  is  preparing  the  recommended  approach  should 
do  so  based  on  training  criteria,  and  when  these  are  complete,  they 
should  be  negotiated  with  appropriate  managers  in  order  to  be  sure  that 
the  best  cost  tradeoffs  can  be  made. 

As  experience  in  IPISD  accumulates,  more  and  better  cost  information 
will  allow  more  cost  effective  decisions  to  be  made. 

Historically,  training  costs  have  typically  been  calculated  on  a 
"tra.ining/man-year  basis."  While  this  was  an  appropriate  procedure  for 
the  more  traditional  forms  of  training,  man/year  costs  do  not  take  into 
account  the  increased  productivity  which  often  results  from  successful 
application  of  ISD.  Future  cost  models  for  ISD-type  courses  must  take 
into  account  the  unit  output  cost  in  contrast  to  the  man/year  costs. 

Regardless  of  the  cost  model  selected  by  your  local  command,  costs 
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ACRONYMS 

AFS  - Air  Force  Speciality 
AR  - Array' Regulations 
■ CMI  - Computer  Managed  Instruction 
CRT  - Criterion  Referenced  Test 

CODAP  - Comprehensive  Occupational  Data  Analysis  Programs 

DOS  - Defense  Occupational  Specialities 

FM  - Field  Manuals 

FOOT  - Formal  On  The  Job  Training 

GF.D  - General  Educational  Development 

HQ  - Headquarters 

. . ^ .1  * - . ... 

ISD  - Instructional  Systems  Development 


! ISS  - Installation  Support  School 

\ ITV  - Instructional  Television 

['  JPA  • Job  Performance  Aids 

JPM  - Job  Performance  Measure 
>,  KOR  - Knowledge  of  Results 


Preceding  page  blank 


151 


152 


LO  - Learning  Objective 
LS  - Learning  Step 

MODB  - Military  Occupational  Data  Bank 

MOS  - Military  Occupational  Specialities  (Army/Marine  Corps) 

MWO  - Modification  Work  Orders 
NIH  - Not  Invented  Here 

NOTAP  - Naval  Occupational  Task  Analysis  Program 

OJT  - On  The  Job  Training 

OSR  - Occupational  Survey  Report 

POI  ••  Program  of  Instruction 

QQPRI  - Qualitative  and  Quantitative  Personnel  Requirements  Identification 

RS  - Resident  School 

SME  - Subject  Matter  Expert 

SMP  - System  Master  Plan 

SOP  - Standing  Operation  Procedures,  Standard  Operating  Procedures 
STEP  - Self-Teaching  Exportable  Package 
TAK  - Trainer  Appraisal  Kit 
TI  - Traditional  Instruction 


TLQ  - Terminal  Learning  Objective 

TM  - Technical  Manuals 

TOE  - Tables  of  Organization  and  Equipment 

TRADOC  - U.S.  Training  and  Doctrine  Command 


GLOSSARY 


ABILITY  GROUPING:  Arrangement  whereby  students  are  assigned  to  groups 
on  the  basis  of  aptitude  testing. 


ABSOLUTE  STANDARDS:  A statement  defining  the  exact  level  of  performance 
required  of  a student  as  a demonstration  that  he  nas  mas- 
tered the  course  objective(s) . Criterion-referenced  tests 
are  usually  based  on  an  absolute  standard. 


ACHIEVEMENT  GROUPING:  Arrangement  whereby  students  are  assigned  to 

groups  according  to  their  performance  on  pretests  of  units 
of  the  course. 


ACTION:  Occurs  in  terminal  learning  objectives  and  learning  objectives; 

describes  the  specific  behavior  the  learner  is  to  exhibit 
after  training. 


ACTION  VERBS:  Verbs  that  convey  action  and  reflect  the  type  of  learn- 

ing that  is  to  occur.  Action  verbs  must  reflect  behaviors 
that  are  measureable,  observable,  verifiable,  and  reliable 


ACTIVITY  STEP:  One  simple  operation  or  movement  that  comprises  part  of 
a job.  A job  performance  standard  consists  of  a list  of 
these  operations  or  movements. 


ADJUNCT  PROGRAMMING:  A method  of  combining  the  features  of  good  exist- 
ing instructional  materials  (e.g.,  films,  textbooks)  with 
special  directions  or  questions  to  guide  the  learner. 


ADMINISTRATIVE  CRITERIA:  In  media  selection,  the  options  that  course- 

ware be  developed  locally  or  at  some  central  location. 


ALGORITHM:  A rule  or  procedure  for  accomplishing  a task  or  solving 
a problem. 


ALPHANUMERIC:  Refers  to  a combination  of  letters  and  numbers;  for 

example,  on  the  keyboard  of  a teletype. 


ALTERNATE  PATH:  Refers  to  elements  which  have  relationships  in  which 

the  specific  situation  encountered  determines  the  appro- 
priate sequence,  or  it  may  be  another  way  of  meeting  the 
same  objective. 
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ASSESSMENT:  A judgment  of  the  effectiveness  and  efficiency  of  a 

training  system,  In  terns  of  measurement  and  evaluation. 


ASSOCIATION  DEVICES:  Memory  aids,  techniques  which  ease  recall. 
Mnemonic  devices. 


ATTITUDE:  A persisting  state  of  a person  that  Influences  his  choice 
of  action. 


ATTITUDE  MEASURE:  An  instrument,  designed  to  gather  information  about 
how  people  feel  toward  a particular  object.  This  could 
Include  liking  or  disliking  sitject  matter,  usefulness 
of  a medium,  or  opinions  about  the  medium. 


AUDIO-ONLY  PROGRAM:  A production  which  does  not  contain  any  video 
or  pictures;  for  example,  a record  or  radio  program. 


AUDIO  PRODUCER:  Prepares  tape  recordings  and  produces  audio  programs 

The  audio  producer  combines  narration,  music,  and 
other  sound  effects  in  the  production  of  an  audio  pro- 
gram. 

AUDIOVISUAL  MEDIA:  Refers  to  any  device  such  as  television  or  film 

which  is  both  seen  and  heard. 


BASELINE  DATA:  Valid  and  reliable  information  about  the  current 

level  of  performance  of  the  Intended  student  population. 
This  data  can  be  used  tc  confirm  the  need  to  develop  new 
instruction,  or  can  be  used  as  a comparison  in  ascer- 
taining differences  between  students’  performance  be- 
fore and  after  instruction. 


BEHAVIORAL  ATTRIBUTES:  Qualities  or  activities  that  characterize  an 
object  or  process.  Behavioral  attributes  characterize 
each  category  of  learning. 


BLOCK  SCHEDULING:  Mode  of  instruction  whereby  ail  students  receive 
the  same  instruction  at  the  same  time. 


BLOCKING: 


Refers  tc  the  process  of  defining  and  Illustrating  the 
different  camera  movements  and  camera  shots  In  a tele- 
vision or  film  script.  A blocked  script  may  also 
contain  directions  as  to  the  movement  of  actors  as  well 
as  scenery  changes. 


CHECKLIST:  Job  performance  aid  which  lists  the  elements  of  a task 
inthe  sequence  of  execution.  The  job  holder  places 
a check  beside  each  element  as  it  is  accomplished,  thus 
Insuring  that  the  task  is  completed. 


CHRONOLOGICAL  ORDER:  Arranging  content  in  order  from  one  topic  to 
another  based  on  when  they  occurred  in  time. 


COMMON-FACTOR  LEARNING  OBJECTIVES:  Refers  to  learning  objectives  that 
are  identical,  or  that  have  identical  action  words  and 
similar  objects  of  the  action  in  the  learning  objective 
statement. 


COMPARATIVE  SEQUENCE:  Sequencing  which  starts  with  familiar  topics 

and  goes  to  unfamiliar  ones. 


COMPLEXITY  CRITERION:  In  media  selection,  the  degree  of  complexity 

required  of  Instructional  materials  in  order  to  ade- 
quately train  students  to  meet  learning  objectives. 


COMPUTER  MODELS  TECHNIQUE:  Occurs  during  the  simulation  of  an  opera- 
tional system;  involves  having  a computer  simulate  the 
major  operations  of  the  system,  under  a variety  of  condi- 
tions. 


CONDITIONS:  Occurs  in  terminal  learning  objectives;  describes  what 
is  presented  to  the  student  in  order  to  accomplish  the 
specified  action,  that  is,  it  describes  the  important 
aspects  of  the  performance  environment. 


CONTIGUITY:  Refers,  in  learning,  to  the  principle  that  events  which 

occur  closely  together  become  associated  by  the  learner. 


CONTINGENCY  MANAGEMENT:  The  establishment  of  a set  of  procedures  by 

which  trainees  are  required  to  perform  a certain  amount  of 
work  or  to  achieve  certain  objectives  before  engaging  in 
activities  that  are  preferred  by  the  trainee  (e.g.,  recrea- 
tion, a break,  or  a more  desirable  training  event). 
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COURSE  DOCUMENTATION:  Information  describing  the  current  content 

of  a course  (Instructional  materials,  tests,  Instructor's 
manual,  evaluation  plan,  student's  manual)  and  Its  develop- 
mental history  (job  analysis,  criteria  for  selecting  tasks 
for  training,  previous  revisions). 


CLUSTERING:  A process  of  organizing  many  tasks  Into  yrcups  for  the 
purpose  of  deciding  upon  the  optimal  instructional  set- 
ting mix  for  that  group  of  tasks. 


CRITERION-REFERENCED  TEST:  Measures  what  an  individual  can  do  or 
knows,  compared  to  what  he  must  be  able  to  do  or  must 
know  in  order  to  successfully  perform  a task,  Hero  an 
Individual's  performance  is  compared  to  external  cri- 
teria or  performance  standards  which  are  derived  from  an 
analysis  of  what  is  required  to  do  a particular  task. 


CRITICAL  CUE:  Cue  which  mush  be  correctly  interpreted  by  the  student  be- 

fore we  can  correctly  perform  the  associated  task. 


CRITICAL  SEQUENCE.  Sequencing  of  topics  or  objectives  according  to 
their  importance. 


CUE:  A word  or  other  signal  that  Initiates  or  guides  behavior;  a 

prompt. 


CUT-OFF  SCORE:  Minimum  passing  score. 


DATA:  Collection  of  facts  or  numerical  values  resulting  from  observa- 

tions of  situations,  objects,  or  people. 


DATA  COLLECTION  PLAN:.  An  outline  of  the  procedures  and  techniques 
that  will  be  used  to  gather  information  for  any  specific 
purpose. 


DATA  RECORDING  PLAN:  Method  of  tabulating  background  responses  and 
test  data. 


DECAY  RATE:  The  amount  of  time  it  takes  a trainee  to  forget  what  he 
has  learned  in  school.  If  the  decay  rate  is  high  then  a 
trainee  should  not  receive  instruction  in  a specific 
task  until  shortly  before  he  will  actually  perform  it. 
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DECISION  TREE:  Flowchart;  graphic  representation  of  the  sequence  of  a 
specific  activity  or  operation. 


DELIVERY  SYSTEM:  Any  method  containing  plans  and  procedures  for  the 
presentation  of  instruction.  Platform  instruction,  tele- 
vision, FQJT,  and  STEPs  are  all  delivery  systems. 


DEPENDENT  RELATIONSHIP:  Occurs  when  skills  and  knowledges  In  one 
learning  objective  are  closely  related  to  those  in  the 
other  learning  objective.  In  order  to  master  one> of  the 
learning  objectives,  it  is  first  necessary  to  learn  the 
other. 


DOWNTIME:  Refers  to  the  period  of  time  when  equipment  is  inoperable. 


DUTY:  One  of  the  major  subdivisions  of  work  performed  by  one  indi- 
vidual. One  or  more  duties  constitute  a job. 


DUTY  TITLE:  Categorizes  groups  of  tasks  under  identifiable  head- 

ings to  Sielp  in  the  organizing  of  lists  of  tasks. 


EMPIRICALLY  BASED  REVISION:  Revision  based  on  the  results  of  test 
data  and  the  collection  of  other  types  of  quantitative 
Information. 


ENTPY  BEHAVIOR:  The  skill,  knowledge,  and/or  attitude  required  be- 
fore beginning  a new  segment  of  instruction;  also  may 
refer  to  the  capability  a person  has  prior  to  new 
learning . 


ENTRY  SKILLS:  Specific,  measurable  behaviors  that  have  been  determined 
through  the  process  of  analysis  of  learning  require- 
ments to  be-. basic  to  subsequent  knowledge  cr  skill  in  the 
course. 


ENTRY  SKIl LS  TEST:  A measurement  instrument  designed  to  determine  if 
a student  already  possesses  certain  skills  or  knowledge 
needed  as  a prerequisite  before  undertaking  new  instruc- 
tion. 


ENTRY  TEST:  Contains  items  based  on  the  objectives  that  the  intended 
students  must  have  mastered  in  order  to  begin  the  course. 
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ERROR  OF  HALO:  Occurs  when  an  observer  sometimes  allows  his  rating 
of  performance  to  be  Influenced  by  his  general  impres- 
sion of  a person. 


ERRORS  OF  LOGIC:  Occur  when  two  or  more  traits  are  being  rated.  It 
is  present  if  an  observer  tends  to  give  similar  ratings 
to  traits  which  do  not  necessarily  go  together.  The 
traits  are  related  only  in  the  mind  of  the  person  making 
the  error. 


ERRORS  OF  STANDARD:  Occur  when  observers  tend  to  rate  performers  too 
high  or  too  low  because  of  differences  in  their  standards. 


EVALUATION:  The  process  of  interpreting  the  results  of  measurement 
data  (e.g.,  tests,  JPMs)  for  the  purpose  of  making  a 
judgment  or  decision  on  the  instruction  or  on  the  success 
of  a trainee. 


EVALUATION  CRITERIA:  The  measures  used  to  determine  the  adequacy 
of  performance. 


EVALUATION  PLAN:  A method  or  outline  of  what  set  of  procedures  will 
be  used  to  gather  data  and  information  for  the  purpose 
of  assessing  a course  of  instruction. 


EXTERNAL  CUES:  Signals  for  action  that  exist  outside  of  the  student 

(conditions,  features,  or  characteristics  of  the  job 
environment  that  trigger  action). 


FALSE  NEGATIVE:  Occurs  when  a person  can  perform  the  task  but 
receives  a failing  score  on  the  test. 


FALSE  POSITIVE:  Occurs  when  a person  cannot  perform  the  task  but 
receives  a passing  score  on  the  test. 


FEEDBACK:  The  return  of  information.  Information  on  student  per- 
formance is  "fed"  back  to  the  student  so  that  he  can 
improve  that  performance;  to  the  instructional  designer 
so  that  he  can  Improve  materials  and  procedures  on  the 
b?.sis  of  student  needs;  to  the  management  system  so  it 
can  monitor  the  internal  and  external  integrity  of  the 
instruction  and  make  appropriate  revisions.  Or,  refers 
to  the  flow  of  data  or  information  from  one  step  in 
the  ISD  Model  to  others. 
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FOOT— FORMAL  ON-THE- JOB  TRAINING:  This  type  of  training  takes  place 
in  the  actual  work  situation. 


FOLLOW-UP  ACTIVITIES:  The  work  events  that  occur  after  a course  of 
instruction  has  been  completed. 


FORMATIVE  EVALUATION:  The  iterative  process  of  developing  and 
improving  Instructional  materials  and  procedures. 


FIDELITY:  Refers  to  how  well  the  actions,  conditions,  cues,  and 
standards  of  the  JPM  approximate  those  of  the  task. 


FIELD  USER  NEEDS:  The  general  and  specific  duties  that  will  have 
to  be  taught  to  the  trainee  if  he  is  to  be  able  to 
adequately  perform  in  a real  world  environment. 


FIRST  DRAFT  MATERIALS:  Any  materials  (book,  film,  etc.)  which  are 
not  yet  committed  to  their  final  form.  First  draft 
refers  to  the  fact  that  the  materials  are  still  in 
'rough*  form  and  will  be  revised  on  the  basis  of  test 
results  and  other  data. 


FLOWCHART:  A graphic  representation  of  the  sequence  of  a specific 
activity  or  operation;  decision  tree. 


FRONT  END  ANALYSIS:  Refers  to  job  analysis,  selection  of  tasks  for 
training,  and  development  of  JPMs. 


FIXED  SEQUENCE:  Refers  to  elements  that  are  always  done  ir  the  same 

order. 


GRAPHIC  ARTIST:  Designs  and  prepares  a wide  variety  of  visual  illu- 
strations such  as  graphs,  charts,  and  diagrams. 


GRAPHIC  SCALE:  Measurement  device  which  includes  some  type  of  number 
line  on  which  students  indicate  their  attitude  toward 
a social  object. 


GO  NO-GO:  Pass-fail;  criterion  of  evaluation  whereby  student  can 
not  be  “partially  correct".  He  is  either  100?.  correct 
(go)  or  1ncorrc:t  (no-go). 
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GROUP  MANAGEMENT  PLAN:  Arrangement  whereby  instruction  is  scheduled 
and  conducted  for  groups  instead  of  individuals. 


GROUP  TRAINING:  A group  of  people  gathered  together  for  the  purpose 
of  receiving  information  or  instruction  in  the  performance 
of  some  specific  task. 


HARD  DATA:  A direct  and  precise  measure  of  a specific  performance. 

A JPM  is  an  example  of  hard  data  while  an  attitude  question- 
naire is  a less  direct  measure,  providing  soft  data. 


HIGH  DENSITY  SIGNAL:  A signal  containing  many  cues.  A low  density 

signal  contains  few  cues. 


INDEPENDENT  RELATIONSHIP:  Occurs  when  skills  and  knowledges  in  one 

objective  are  unrelated  to  those  in  the  other  objective. 
Mastering  one  of  the  objectives  does  not  simplify  the 
other. 


INDICATOR  BEHAVIOR:  Refers  to  that  behavior  that  indicates  the 

presence  of  a specific  attitude. 


INDIVIDUALIZED  INSTRUCTION;  Refers,  in  the  ISD  Model,  tv  - management 
scheme  which  permits  individual  characteristics  of 
trainees  to  be  a major  determinant  of  the  kind  ana  amount 
of  instruction  given.  Here,  it  nearly  always  implies 
some  form  of  self-pacing. 


INSTALLATION  SUPPORT  SCHOOLS:  Organized  and  operated  by  individual 

units  or  commands  to  meet  local  training  requirements. 


INSTRUCTIONAL  CONDITIONS:  The  amount  of  participator  which  the 

instruction  requires  of  the  learner.  Instructional 
conditions  may  be  active  (the  learner  produces  or  prac- 
tices) or  passive  (the  learner  sits  and  listens). 


INSTRUCTIONAL  DESIGNER:  Person  who  designs  and  develops  a program  or 

course  of  stucies  based  on  a systematic  analysis. 


INFORMATION:  Knowledge;  the  facts,  names,  labels,  and  larger  bodies 

of  knowledge  that  are  necessary  for  successful  job 
performance. 


INSTRUCTIONAL  MANAGEMENT  PLAN:  The  specifications  for  the  scheduling, 
instruction  and  evaluation  of  trainees  toward  the  goal  of 
course  completion. 

INSTRUCTIONAL  PROGRAM:  The  development  of  various  materials  (books, 
audiovisual  productions,  etc.)  designed  to  achieve  a 
specific  training  goal. 


INSTRUCTIONAL  SETTING:  The  vehicle  through  which  a trainee  who 

Initially  is  not  able  to  perform  a task  becomes  profi- 
cient in  performing  the  task;  for  example,  performance 
aids,  self-teaching  exportable  packages,  formal  on-lob 
training,  installation  support  schools,  and  resident 
schools. 


INSTRUCTIONAL  SUPPORT:  Learning  resources;  different  kinds  of 

material,  number  of  Instructors,  amount  of  time,  etc. 
which  will  contribute  to  the  learning  situation. 


INSTRUCTIONAL  SYSTEM:  The  total  effort,  distinct  from  the  operating 

system  by  location,  authority,  or  mission,  that  is  con- 
cerned with  the  preparation  of  individuals  to  serve  the 
operating  system. 


INTERNAL  CUES:  Internal  biological  "ignals  that  initiate  or  guide 

behavior. 


INTERNAL  EVALUATION:  Assessment  of  the  effectiveness  of  an  instruc- 
tional program  in  terms  of  student  performance  on 
stated  terminal  learning  objectives. 


JOb:  The  duties  and  tasks  performed  by  a single  worker  constitute 
his  job.  If  identical  duties  and  tasks  are  performed  by 
several  individuals,  they  all  hold  the  same  job.  The 
job  is  the  basic  unit  used  in  carrying  out  the  personnel 
actions  of  selection,  training,  classification,  and 
assignment. 


JOB  ANALYSIS:  The  basic  method  used  to  obtain  a detailed  listings  of 

duites,  tasks,  and  elements  necessary  to  perform  a 
clearly  defined,  specific  job,  involving  observations  of^.. 
workers  and  conversations  with  those  who  know  the  job, 

■in  order  to  describe  in  detail  the  work  involved,  includ- 
ing conditions  and  standards. 
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JOB  FIDEL1  Y:  The  degree  to  which  a testing  situation  truthfully 
and  accurately  reflects  the  job  situation. 


JOB  PERFORMANCF  MEASURES:  Tests  that  are  used  to  evaluate  proficiency 
of  a job  holder  or.  each  task  he  performs. 


JON  PERFORMANCE  TEST:  Test  used  to  determine  whether  or  how  well  an 
individual  can  perform  a job.  In  may  include  oither  all 
of  the  job  performance  measures  for  a particular  job  or 
a subset  of  the  job  performance  measures. 


JPA--J0B  PERFORMANCE  AID:  A checklist,  instruction  sheet,  or  other 
device  that  offers  a possible  alternative  to  training 
rather  than  an  actual  method  of  training;  they  are  de- 
veloped to  eliminate  or  minimize  training  requirements 
for  some  tasks. 


KNOWLEDGE  OF  RESULTS:  Feedback;  information  provided  to  the  student 
indicating  the  correctness  of  his  response.  Evaluative 
knowledge  of  results  indicates  what  a student  is  doing 
right  and  what  he  is  doing  wrong.  Comparative  knowledge 
of  results  indicates  how  the  student's  response  compares 
to  the  objective  or  standard  established  by  the  instruc- 
tor. 


LEARNER  CHARACTERICTICS:  The  traits  possessed  by  learners  that  could 
affect  their  ability  to  learn  (e.g.,  age,  I.Q.,  reading 
level,  etc.). 


LEARNING  ACTIVITY:  The  specific  behaviors  a student  performs  during 
a particular  episode  of  learning. 


LEARNING  ANALYSIS:  A procedure  to  identify  subelementc  that  must  be 
learned  before  a person  can  achieve  mastery  of  the 
performance . 


LEARNING  CATEGORY:  A division  of  learning  behavior.  All  learning 
may  be  classified  into  one  of  four  learning  categories: 
mental  skill,  physical  skill,  information,  or  attitude. 


LEARNING  EVENT:  The  immediate  outcome  of  a learning  activity. 


LEARNING  GUIDELINES:  Statements  which  specify  the  learning  events 
and  activities  appropriate  to  specific  Instruction. 
Learning  guidelines  combine  to  form  learning  sub- 
categories. 


LEARNING  HIERARCHY:  Graphically  portrays  the  relationships  ?<mnng. 

learning  tasks  In  which  some  tasks  must  be  mastered 
before  others  can  be  learned. 


LEARNING  OBJECTIVE:  Describes  precisely  what  Is  to  be  learned  in 
terms  of  the  expected  student  performance  under  speci- 
fied conditions  to  accepted  standards.  These  learning 
objectives  identify  the  mental  skills,  information, 
attitudes,  or  physical  skills  that  are  required  to  per- 
form the  terminal  learning  objective. 


LEARNING  RESOURCE  CENTER:  Library  containing  Instructional  materials 
and  areas  for  viewing  and  study. 


LEARNING  STEP:  Occurs  when  learning  objectives  ere  broken  down  into 

smaller  parts. 


LEARNING  SUB-CATEGORY:  A division  of  a learning  category. 

LEARNING  TASK  ANALYSIS:  Procedure  used  in  the  domain  of  intellectual 
skills  to  identify  prerequisite  tasks  that  must  bt 
learned  before  a person  can  learn  a giver,  task. 


LINK  TRAINER:  Mechanical  training  device  which  simulates  the  cock- 
pit of  an  aircraft. 


RESPONSE  BIAS:  Tendency  to  favor  a certain  response  over  others. 


MANAGEMENT  PLAN:  Program  for  the  assignment,  monitoring,  and  assess- 
ment of  the  personnel,  materials,  and  resources  dedi- 
cated to  a specific  mission,  operation,  oi'  function. 


MASTERY:  In  terms  of  learning,  refers  to  meeting  all  of  the  specified 
minimum  requirements  for  a specific  performance.  Criteria 
for  mastery  are  defined  in  the  design  phase  of  the  ISD 
Model . 
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MEAN:  Arithmetic  average  calculated  by  adding  up  all  scores  and 
dividing  by  the  number  of, scores. 


MEASUREMENT:  Consists  of  rules  for  assigning  numbers  to  objects  to 
represent  quantities  of  attributes. 

MEASUREMENT  ERRORS:  Incorrect  procedures  carried  out  during  the 

measurement  process  which  Invalidate  the  results.  These 
errors  result  from  unfounded  assumtlons  made  by  judges 
or  raters.  ' • '•••  . 


MEASUREMENT  PROCESS:  The  operations  Involved  in  determining  the 

amount  of  an  attribute  (e.g.,  skill,  knowledge,  or  atti- 
tude) possessed  by  a student. 

MEDIA:  Means  for  presenting  instructional  material  to  learners;  for 
example,  books,  audiotapes,  and  filmstrips. 


MEDIA  ALTERNATIVE:  A form  of  instructional  material  that  contains 
the  stimulus  criteria  required  by  a specific  learning 
activity; 


MEDIA  MIX:  Combination. of  different  media  used  to  present  a unit  of 

instruction. 

MEDIA  POOL:  All  of  the  media  options  suitable  for  a given  unit  of 
instruction.  The  final  media  choice  is  drawn  from  the 
media  pool . 


MEDIA  SELECTION:  Is  the  major  means  of  determining  how  instruction  is 

to  he  packaged  and  presented  to  the  student. 


MENTAL  SET:  A preparatory  mental  adjustment,  or  readiness,  for  a 
particular  type  of  experience. 

MENIAL  SKILLS:  Those  processes  of  identifying,  classifying,  using 
rules,  and  solving  problems  that  involve  active  mental 
processing.,  Mental  skills  imply  the  capability  of 
applying  the  learning  to  some  situation  and  demonstrating 
the  mental  skill,  such  as  thinking,  creating,  and 
analyzing. 
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MNEMONICS:  Methods  which  make  information  easier  to  remember;  memory 
aids. 


MODE  OF  INSTRUCTION:  Method  of  scheduling  materials  presentation.  The 
Instructional  mode  r«.y  be  Individualized  (self-pacing)  or 
group  (block  scheduling). 


MODULE:  An  individualized  self-instructional  package  usually  con- 
taining all  the  necessary  materials  a learner  needs  to 
meet  some  or  part  of  a terminal  '.earning  objective. 


MULTIMEDIA  PACKAGE:  Self-contained  instructional  unit  in  more  than 
one  medium. 


NARRATION:  Is  the  voice  overheard  on  an  audiovisual  program. 


NARRATOR:  Is  the  person  whose  voice  is  hea>”d  describing  or  commenting 

upon  the  content  of  a film,  television  program,  etc. 


NUMERICAL  SCALE:  Measurement  device  which  associates  verbal  descrip- 
tions of  social  objects  with  numbers  and  requires  students 
to  indicate  their  attitudes  by  marking  the  appropriate 
number.  • 


OBSERVATION  INTERVIEW:  Job  holder  is  observed  in  the  job  environment 
performing  all  or  a substantial  part  of  the  job;  the  job 
holder  performs  the  job  while  the  analyst  ask  questions  I 


OFF-LINE:  Refers  to  any  activity  which  does  not  take  place  as  part 
of  the  regular  production  process. 


OVERLEARNING:  Refers  to  the  continual  practice  on  a learning  task  by  a 
person  who  has  correctly  performed  the  task. 


PEER  TUTORING:  A form  of  instruction  in  which  students  at  the  same 
or  more  advanced  level  of  knowledge  provide  instruction 
to  students  at  the  same  or  lower  level  of  knowledge  on 
the  specific  objectives  under  consideration.  Peer  tutors 
are  not  members  of  the  existing  instructional  establish- 
ment. 
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PERFORMANCE  EVALUATION:  The  gathering  of  data  to  specifically 

determine  the  success  of  students  on  a specific  task, 
as  a result  of  n training  program. 


PERFORMANCE  MEASURES:  The  aosolute  standard  by  which  a job  perfor- 

mance Is  judged.  A performance  measure  Is  the  inven- 
tory of  job  tasks  with  each  performance  objective. 


PERSE  ERATE:  Continue  an  activity  until  it  is  completed,  regardless 
of  the  difficulty,  or  the  appropriateness  of  the  solu- 
tion technique  to  the  problem. 


PERT -PROGRAM  EVALUATION  REVIEW  TECHNIQUE:  PERT  is  a method  of  moni- 
toring the  flow  of  a large  project  by  breaking  it  down 
into  small  individual  activities  and  assigning  each 
activity  a specified  amount  of  time  for  completion. 


PHYSICAL  SKILLS:  Specified  muscular  activities  for  accomplishing 
a goal . 


POST  FEEDBACK  DELAY:  The  pause  which  follows  the  presentation  of 
feedback.  This  allows  time  for  the  correct  response  to 
"sink  in." 

POSTTEST:  A test  administered  after  the  completion  of  instruction  to 

assess  whether  a student  has  mastered  the  objectives  of 
the  course  or  unit. 


PREDICTIVE  VALIDITY:  The  ability  of  a test  score  to  accurately  fore- 
cast future  performance. 


PREDIFFERENTIATION  OF  STIMULI:  Pointing  out  the  distinguishing 
features  of  an  object  and  explaining  the  differences 
between  them. 


PRETEST:  Administered  prior  to  instruction  to  determine  how  much 
the  student  already  knows. 


PROCESS  EVALUATION:  An  early  stage  in  ISU  development  that  identifies 
which  steps  in  the  model  will  be  used  for  the  course  under 
development.  The  purpose  of  the  process  evaluation  is  to 
describe  and  document  the  actual  developmental  process  for 
this  particular  instruction. 
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PROCESS  STANDARDS;  Refers  to  the  conditions  which  must  be  satisfied  for 
a job  to  be  successfully  completed.  Process  standards  refer 
to  sequence,  accuracy,  speed  of  performance,  and  complete- 
ness. 


PROGRAMMED  INSTRUCTION:  Instructional  materials  which  present  subject 

matter  In  a series  of  small  sequential  units  which  require 
responses  from  the  student. 


PROMPT:  A word  or  other  signal  that  initiates  or  guides  behavior; 
a cue. 


QUALITY  CONTROL:  Process  of  measuring  and  evaluating  in  order  to  main 
tain  course  standards  through  adjustments  in  instructional 
materials  or  procedures. 


QUALITY  CONTROL  DATA:  Information  which  reflects  the  degree  of  success 

achieved  by  a system  or  operation. 


RANDOM  SELECTION:  Choosing  people  or  objects  at  random  rather  than 

according  to  some  systematic  plan. 


RANK  ORDER:  The  assignment  of  ranks  to  students.  This  could  refer  to 
groups,  such  as  the  top  10X,  or  simply  listing  each  stu- 
dent from  highest  to  lowest. , Rank  ordering  is  appro- 
priate when  there  is  a need  to  select  the  fastest,  the 
most  accurate,  or  the  best  producer. 


RATING  ERRORS:  Errors  of  standards,  ratio,  and  logic. 


RATING  SCALE:  A measurement  device  in  which  a student  must  choose  a 
response  from  a range  of  choices  arranged  in  a continuum 
from  low  to  high  or  good  to  bad,  etc. 


REGULATIONS:  Rules  for  appropriate  conduct  and  behavior. 

RELIABILITY:  The  consistency  with  which  a test  measures  the  amount  of 
student  achievement. 


RESIDENT  SCHOOLS:  These  schools  are  designed  to  meet  service-wide 
training  requirements. 
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REVISION  PLAN:  A detailed  outline  of  the  procedures  to  be  taken 
to  modify  the  structure  or  content  of  a course. 


REWARD  SEQUENCE:  Scheduling  the  more  pleasant  activity  to  follow 
the  less  pleasant  activity;  can  be  used  to  provide 
a reward  for  completion. 


SAMPLE:  A portion  or  small  segment  of  the  students  for  whom  instruc 
tion  is  designed. 


SAMPLING  PLAN:  Procedure  for  selecting  a small  but  representative 
group  from  a larger  population. 


SCALE:  In  media  selection,  some  materials  must  represent  actual 

objects  and  accurately  represent  the  dimensions  of 
those  objects.  A model  may,  for  example,  be  full 
scale,  half  scale,  or  on  a 1 to  10  scale  with  the 
actual  object. 


SELF  PACING:  Mode  of  instruction  whereby  each  student  works  through 

the  instructional  materials  at  his  own  ►'ate  of  speed. 


SELF-PACED  MANAGEMENT  PLAN:  Arranganent  whereby  instruction  is 

scheduled  and  conducted  for  individual  students  rather 
than  groups  of  students. 


SELF-TEACHING  EXPORTABLE  PACKAGES:  Self  instructional  study  units; 

generally  sent  to  the  student  wherever  he  is  stationed. 


SEQUENCING:  Ordering  instruction;  proper  sequencing  allows  the 

learner  to  make  the  transition  from  one  skill  or  body 
of  knowledge  to  another,  and  assures  that  supporting 
skills  and  knowledge  are  acquired  before  dependent 
performances  are  introduced. 


SHAPING:  Gradually  changing  a student’s  behavior  until  it  is  correct. 

SIGNAL:  Cue  that  initiates  and  directs  activity. 
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SIMULATION:  Any  change  from  reality  or  any  imitation  of  reality. 

Three  types  are  common:  simulating  part  of  the  system, 
simulating  the  operation  of  the  system,  and  simulating 
the  environment  in  which  the  system  will  operate. 


SIMULATORS:  Machines  or  processes  designed  to  provide  training  which 
will  have  high  positive  transfer  to  the  real  world  equip- 
ment or  situation.  Simulators  are  ordinarily  cheaper, 
safer,  or  more  available  than  the  actual  situation  or 
equipment. 


SLIDE-TAPE:  A combination  of  visual  slides  ana  an  audio  tape  syn 
chronizea  so  that  the  audio  describes  the  content  of 
the  slides. 


SOFT  DATA:  Obtained  from  attitude  or  opinion  surveys.  This  data  is 
not  as  reliable  as  hard  data. 


STANDARDS:  Occurs  in  terminal  learning  objectives  or  learning 

objectives;  describes  the  criterion  or  standard  of  per- 
formance which  must  be  attained. 


STIMULUS  CRITERIA:  Those  basic  qualities  or  capabilities  of  a 

medium  that  are  required  to  carry  out  the  intent  of  the 
learning  activity;  for  example,  visual  images,  motion, 
color,  and  sound. 


STORYBOARD:  A collection  or  series  of  small  pictures  which  describe 
the  action  and  content  that  will  be  contained  in  an  audio 
visual  or  visual -only  production.  A sequence  of  these 
small  pictures  comprise  a storyboard. 


SUBJECT  MATTER  EXPERT:  A person  who  has  professional  skill  in  the 
performance  of  some  job  and  who  is  consulted  by  an  in- 
structional designer  in  the  process  of  job  task  analysis. 


SUPPORTIVE  RELATIONSHIP:  Occurs  when  skills  and  knowledges  in  one 
objective  have  some  relationship  to  those  in  the  other 
objective;  the  learning  involved  in  mastery  of  one  learn- 
ing objective  transfers  to  the  other,  making  learning 
involved  in  the  mastery  of  the  other  easier. 
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SYMBOL:.  Anything  that  stands  for  or  represents  something  else.  A 
plus  sign  (+)  Is  a symbol  for  the  mathematical  operation 
of  addition. 


SYSTEM  MASTER  PLAN:  Control  document  used  to  coordinate  the  develop- 
ment and  Implementation  of  an  instructional  program. 


SYNCHRONIZING  PULSE:  An  audible  or  inaudible  sound  used  to  coordinate 
the  audio  and  video  portions  of  a slide-tape  program  so 
that  audio  and  video  (i.e.,  slide  and  narration),  are 
coordinated. 


SYSTEMS  APPROACH:  A generic  term  referring  to  the  orderly  process  of 

analysis,  design,  development,  evaluation,  revision,  and 
operation  of  a collection  of  interrelated  elements. 


TALK-THROUGH  TECHNIQUE:  Occurs  during  the  simulation  of  an  operational 

system;  involves  talking  through  each  operation  in  the 
new  system  to  determine  decisions  and  contingencies. 


TARGET  POPULATION:  The  pool  of  potential  entrants  to  training  for 

which  instructional  materials  are  designed  and  tried  out. 


TASK  DELAY  TOLERANCE:  A measure  of  how  much  delay  can  be  tolerated 
between  the  time  the  need  for  task  performance  becomes 
evident  and  the  time  actual  performance  must  begin. 


TASK:  Formed  in  clusters  which  make  up  duties.  A task  is  the  lowest 
level  cf  behavior  in  a job  that  describes  the  performance 
of  a meaningful  function  in  the  job  under  consideration. 


TASK  INVENTORY:  List  that  itemizes  all  of  the  tasks  that  make  up 
a selected  duty. 


TASK  LEARNING  DIFFICULTY:  Refers  to  time,  effort,  and  assistance  re- 
quired by  a student  to  achieve  performance  proficiency. 


TASK  STANDARD:  A statement  of  how  well  a task  must  be  performed. 
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TASK  STATEMENT:  A statement  of  highly  specific  action  which  has  a 
verb  and  object;  for  example,  sort  mail. 


TECHNICAL  ORDERS:  Military  regulations  which  deal  with  the  specific 

nature  of  technical  materials  and  equipment. 


TERMINAL  LEARNING  OBJECTIVE:,  Derived  from  job  performance  measures, 
TLOs  are  to  be  attained  during  training.  TLOs  are  broken 
down  into  their  component  parts  which  are  documented  as 
learning  objectives  which  may  be  further  divided  into 
learning  steps.  Each  TLO  contains  actions,  conditions, 
and  standards. 


TESTS:  Any  device  or  technique  used  to  measure  the  performance  of  a 
student  on  a specific  task  or  subject  matter. 


TESTING  CONSTRAINTS:  Limitations  such  as  time,  money,  personnel, 

facilities,  and  other  resources,  which  prohibit  job 
performance  measures  from  being  identical  to  the  tasks 
they  measure. 


TRADE-OFFS:  In  any  systematic  approach  to  instruction,  it  is 

necessary  to  make  compromises  between  what  is  desirable 
and  what  is  possible.  Ordinarily,  these  decisions  in- 
volve increases  or  decreases  in  time,  money,  facilities, 
equipment,  or  personnel.  Training  aids  and  simulators 
represent  examples  of  trade-offs. 


TRAINER  APPRAISAL  KIT:  A package  of  instructional  materials 

designed  to  provide  a course  instructor  with 
practice  in  the  preparation,  presentation,  and  valida- 
tion of  instruction. 


TRAINING:  The  teaching  of  job  skills.  It  can  take  a number  of  fonns 

such  as  self-teaching  exportable  packages,  training 
manuals,  individual  learning  packages,  FOJT,  or  group 
training. 


TRAINING  SETTING  CRITERIA:  In  media  selection,  the  options  that 

training  must  be  either  small  group,  large  group,  indi- 
vidualized at  a fixed  location,  or  individualized  inde- 
pendent of  location. 
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TRYOUT:  Practice  test;  the  purpose  is  to  make  the. tryout  as  realistic 
as  possible  by  eliminating  as  many  sources  of  unreliability 
as  possible. 


UNDERTRAIN:  Provide  inadequate  training  that  does  not  prepare  a 
student  to  meet  regular  job  performance  requirements. 


VALIDATION:  A process  through  which  a course  is  revised  until  it 
is  effective  in  rea!1?ing  its  instructional  goal. 


VALIDATION  DOCUMENTATION:  A report  which  describes  in  detail  how 
a specific  course  of  instruction  was  validated  and  for 
what  target  population. 


VALIDATION  PROCESS:  Testing  instructional  materials  on  a sample  of 
the  target  population  to  insure  that  the  materials  are 
effective. 


VALIDITY:  The  degree  to  which  a test  measures  what  it  claims  to 

measure. 


VALUE  ENGINEERING:  Refers  to  the  process  of  designing  equipment 

or  Instruction  to  meet  but  not  exceed  the  required  out- 
comes. Ordinarily,  it  refers  to  the  elimination  of 
features  or  instructional  objectives  that  have  not  been 
demonstrated  to  be  positively  necessary. 


VIGILANCE  LEVEL:  General  degree  of  watchfulness  or  attentiveness 
to  what  may  come. 


VISUAL  FORM:  In  media  selection,  refers  to  whether  alphanumeric  or 

pictorial  characteristics  are  required  in  a learning 
situation. 


VISUAL  SPECTRUM:  The  type  of  color  required  of  instructional 

materials.  Some  must  be  with  full  color,  others  may 
be  with  black  and  white  or  shades  of  grey. 


WITHIN-CCURSE  TESTS:  Administered  during  a course  of  instruction 
to  assure  that  all  students  are  "keeping  up”  with  the 
learning  objectives. 
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WORK  ELEMENTS:  The  element  Is  the  smallest  component  In  the 

structure  of  a job.  Elements  combine  to  form  a task, 
tasks  combine  to  form  a duty,  and  duties  combine  to 
form  a job. 
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Analysis  Program  (CODAP), 

I:  121 


Computer- assisted  instruction  (CAI), 
III:  191 

Computer-managed  instruction,  III: 
191-194 

Conditions,  JPM,  I:  177-173 

on-the-job,  task,  I:  24-26,  89-93 
Contiguity,  III:  41 
Contingency-managed  instruction , 

III:  187-189,  215 

Contract,  student  (or  performance), 
III:  183 

Cues,  JPM,  I;  179 

prompts  for  student  learning,  III: 
6-8 

ttsk,  I:  25,  27-28 


Data  collection, 

for  external  evaluation,  V:  68-77 
for  internal  evaluation,  V:  1-8, 
29-46,  53-62 
for  JPMs , 1:45-46 
for  rating  tasks,  I:  133-139 
Data  collection  plan  for  job  analysis, 
alternate  or  short-cut,  I:  47, 

77-78 

form  preparation,  I:  49-51,  59 
group  interviews,  I:  40 
individual  and  observation  inter- 
views, I:  36-37,  41-46,  52-62 
jury-of-experts , I:  39-40 
questionnaire  survey,  I:  37-39, 

45,  46,  68-73 

Decision-making:  learning  sub-category 
5,  III:  46-48,  83-86 
Decisions,  management,  Executive 
Summary:  20,  24-25,  30-152,  36-37, 

4 43,  51  , 54-55,  59,  63,  69-70, 

75-76,  81-82,  85-36,  91-92,  96-97, 
100-101,  108-109,  114-115,  119-120 
Decision  trees,  see  Algorithms 
Defense  Occupational  Specialties  (DOS), 
I:  3,  8,  ?1 4 

Delay  tolerance,  see  Task  delay  ... 


Preceding  page  blank 
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Del  Ivory  system,  III:  713-214, 
see  al so  Media  and  Instructional 
management  plan  and  delivery 
system 

Detecting:  learning  sub-category  4, 
III:  43-45,  78-82 
Duty,  1:  11-12 


Electronic  technician  (production 
personnel),  III:  ,229 
Element(s),  I:  17-22 

alternate-path,  I:  22-24 
fixed-sequence,  I:  22-24 
Entry  behavior,  student,  II:  62-77 
definition  of,  II:  62-63 
entry  tests  for,  II:  70-72 
outputs,  11:76 

procedures  for  determining,  II: 
64-75 

purpose  of  establishing,  II: 
62-63 

protesting,  II:  62,  73-75 
remedial  or  preparatory 
instruction,  II:  69 
revise/test  procedure,  II:  69 
tests  to  verify  assumptions 
about,  II:  63-69 

Entry  test,  development  of,  II:  72 
need  for,  II:  70-72 
Equipment-oriented  jobs,  I:  81-83 
Errors  in  measuring  and  testing, 

I:  201-205;  IV:  33 
of  halo,  I:  203-204;  IV:  33 
of  logic,  I:  204-205 
of  standard,  I:  202;  IV:  33 
Evaluation,  external,  see  External 
evaluation 

Evaluation,  internal,  see  Internal 
evaluati  on 

Existing  courses,  analysis  of,  I; 
213-229 

development  documentation  for, 

I:  220 

evaluating  existing  job  analy- 
ses for,  I:  215-216,  221-222 
evaluating  existing  Job  Perform- 
ance Measures  for,  1:  216, 
225-226 

evaluating  tasks,  I:  223  224 
evaluating  validation  documen- 
tation for,  I:  226 
front  End  Analysis,  I:  216-218 


Existing  courses  (continued) 
outputs,  I:  21871?27:228 
procedures  for,  I:  218-227 
rationale  for,  I:  213,  215 
revisions  indicated  by,  I:  227 
Existing  materials,  review  of,  III: 
198-220 

collection  process  for.  III: 

202-203 

modifications  determined  by,  III: 
206-208,  212-213,  216 
outputs,  III:  216-217 
procedures  for,  III:  201-216 
rationale  for,  III:  198-201 
with  reference  to  delivery  system, 
III:  213-214 

with  reference  to  learner  charac- 
teristics, III:  203-205 
with  reference  to  learning  guide- 
lines, III:  208-213 
with  reference  to  management  plan, 
III:  214-215 

External  evaluation,  V:  63-86 

baseline  data  collection,  V:  70-71 
data  collection  methods,  V:  68-69 
data  collection  procedures,  V:  69- 
77 

data  consolidation,  V:  78-81 
data  requirements,  V:  66-68 
data  sources,  V:  65-66 
documentation,  V:  84-85 
job  performance  evaluation  data 
collection,  V:  71-73 
outputs,  V:  83-85 
personal  interviewdata,  V:  74-75 
planning  phase,  V:  65-69 
procedures,  V:  65-83 
purposes  of,  V:  63-65 
questionnaire  use,  V:  73-74 
recommendations,  V:  81-83 
records  of  students'  performance 
during  instruction,  V:  76-77 
report  (EXER),  V:  78,  83 
time  considerations,  V:  68 
External  Evaluation  Report  (EXER),  V: 
78,  83,  90-91,  93,  95-96,  105,  109- 
110,  112 


Feedback,  111:  6,  9-10 
Formal  On-the-Job  Training  (FflJT),  I: 
231  , 237-238,  256-258;  III:  1 39- 
141  , 260-262 
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Formative  evaluation,  III:  280-281 
Front  End  Analysis,  I:  216-218 


Graphic  artist  (production  person- 
nel), III:  227-228 


Hard  data,  V:  4 


Identifying  symbols,  learning  sub- 
category 3,  III:  41-42,  76-77 
Installation  Support  Schools  (ISS), 
I:  238,  258-259;  III:  140-142 
Instruction,  conducting,  IV:  29-39 
course  documentation,  pre- 
instruction review  of,  IV: 
32-34 

changes  in,  IV:  36-38 
documentation  of  changes, 
problems  for,  IV:  35-37 
follow-up  activities,  IV:  37-38 
function  of  instructor,  IV:  29- 
31,  32-38 
■ outputs,  IV:  38 
procedures,  IV:  32-38 
test  administration,  IV:  32-33, 
34-35 

training  of  instructors,  IV:  34- 
35 

Instruction,  development  of.  III: 
221-287 

adjunct  programs,  III:  251 
aims,  III:  221-223 
audio-only  script  guidelines, 
III:  230-232,  233 
audio-visual  production,  III: 
235-244 

Formal  On-the-Job  Training 
(FOJT) , III:  260-262 
Instructor's  Guide,  III:  265- 
266 

Job  Performance  Aids  (JPAs), 

III:  252-260 

needs  and  constraints,  III: 
224-225 

outputs , III:  270 
platform  lectures,  III:  248 
pre-testing  of  draft  materials, 
III:  263-265 

printed  materials,  III:  246 


Instruction,  development  of  (con- 
tinued) 

procedures,  III:  224-269 
production  personnel.  III:  227-229 
programmed  instruction,  III:  246- 
247 

resources,  III:  225-230 
Self-Teaching  Exportable  Packages 
(STEPs),  III:  248-249 
slide-tape  production,  III:  239- 
244 

Students'  Guide,  III:  269 
supplementary  instruction,  III: 
249-250 

television  program  production, 

III:  244-246 

terms  defined,  III:  286-287 
types  of  instruction.  III:  230- 
263 

user  instructions,  III:  265 
video-only  materials,  III:  232. 
234-235 

Instructional  management  plan  and 
delivery  system,  III:  105-197 
block-scheduling  for  group 
instruction,  III:  124-125 
consumables  and  courseware,  III: 
136-137 

course  management.  III:  126-127 
definition,  III:  105 
equipment  and  facilities,  III: 
135-136 

for  instructors,  III:  131-135 
for  student  processing,  III:  127- 
130 

for  support  personnel.  III:  135 
function  of  guidelines,  III:  106- 
107 

]roup  assignment,  methods  of, 

III:  125 

management  guide  I ines  , III:  124- 
137 

media  costs,  III:  112 
media  mixes,  III:  107-118 
media  selection,  III:  118-124 
outputs,  III:  142-143 
procedures  for  specifying,  III: 
107-142 

program  compl eti on , III:  131 
self-pacing  mode,  111:  124,  125- 
126 

System  Master  Plan  (SMP),  III: 

105,  107,  138-142 
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Instructional  management  plan,  imple- 
mentation or , IV : 1-10 
equipment,  IV:  15 
instructor's  manual,  reviewed 
and  supplemented,  IV;  4-5 
materials,  IV;  15 
procedures,  IV:  4-15 
space  allocation,  IV:  14-15 
staff  training,  IV:  7-12 
student  selection,  IV:  13 
student  manuals,  reviewed  and 
supplemented,  IV:  6 
time  allocation,  IV : 14 
time  relationships  within  ISD 
process,  IV:  3-4 
Trainer  Appraisal  Kit  (TAK), 

IV:  7-12 

Instructional  materials,  validation 
of.  Ill:  280-346 
attitude  measures.  III:  288, 
294-295 

data  analysis  plan.  III:  295-297, 

316 

data  collection  and  recording 
system,  III:  287-293,  314-316 
data  consolidating  and  analyzing, 
III:  304-308,  331 
group  trial,  administering.  III: 
328 

group  trial  procedures,  III:  316- 

317 

individual  trials,  plan  for  con- 
ducting instruction  for.  III: 
297-298 

individual  trials,  steps  in  con- 
ducting, III:  298-303 
learning  objective  rating,  III: 
317-327 

nature  and  purpose  of,  III:  280- 
285 

outputs,  III:  336-339 
participant  selection  for  group 
trials,  III:  317,  328 
plan  for  group  trials,  III:  312- 
328 

plan  for  individual  trials,  III: 
285-298 

platform  instruction.  Ill:  329- 
331 

procedures,  III:  285-336 
revision  of  attitude  measures, 

III:  316 


Instructional  materials,  validation 
of,  (continued) 

r e v i s 1 o n > eq u i renien t s , III:  308- 
311,  332-336 

standards  and  constraints  for 
group  trials,  III:  313-314 
Instructional  methods  and  devices, 

III:  163-172 

audio-only  systems,  III:  163 
audio-visual  systems,  III:  163-166 
computer  simulation,  III:  166-168 
print  materials.  III:  169-171 
special  and  non-standard  items, 

III:  171-172 

visual-only  systems,  III:  168-169 
Instructional  setting(s),  selection 
of,  I:  230-264 

assignment  of  tasks,  I:  233-263 
clustering  tasks,  I:  244-245 
cost  analysis  for,  I:  240-242 
definition  of  instructional 
setting,  I:  230-231 
documentation  for,  I:  262-263 
Formal  On-the-Job  Training  (F0JT), 

I:  23!,  237-238,  256-258 
group  training,  I:  239 
inputs,  I:  231-232 
Installation  Support  Schools  (ISS), 
I:  231,  238,  258-259 
Job  Performance  Aids  (JPAs),  1: 

231,  233-235,  236,  242,  245-249 
Job  Performance  Measures  in,  I: 

232 

outputs,  I:  262-263 
personnel  data  factor  in,  I:  249- 
250 

procedures,  I:  242-263 
Resident  Schools  (RS) , I:  231  , 2 38- 
239,  259 

revision  of  task  assignments,  I: 
260-262 

Self-Teaching  Exportable  Packages 
(STEPs),  I:  231 , 235,  237,  250- 
255 

work  experience,  I:  239-240 
Instructional  Systems  Development  (ISD) 
program,  documentation  for,  I: 
83-84 , 87 

initiation  of,  I:  1-5 
purpose  of  , 1 : 2 , 214 
rationale  for  sequence  of,  !:  5-6 
Instructional  television  (ITV),  III: 
194-195 
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Instructor  evaluation  program,  IV: 
7-8 

Instructor-managed  instruction,  III: 
186-187 

Instructor(s) , role  of,  IV:  7-9, 
29-31  ,.  35-38 
training  of,  IV:  7-12 
Instructor's  Guide,  III:  265-268 
Instructor's  Manual,  IV:  4-5 
Internal  evaluation,  IV:  2-4,  D,  9; 
V:  1-62 

conducting  process,  V:  33-34 
data  forms  and  questionnaires 
(examples) , V:  53-62 
data  ratinq  and  measurement.,  V: 

1-8 

data  (general)  display  arid  in- 
terpretation, V:  34-40 
data  sources  and  uses,  V:  5-8 
documentation,  V:  49-52 
evaluation  designs  (additional), 
V:  46-49 

functions  and  requirements  of, 

V:  1-10 

ISD  documentation  tor  process 
evaluation,  V:  17-19 
outputs,  V:  49 

performance  data  display  and 
interpretation,  V:  40-46 
Performance  Evaluation  Plan,  V: 
22-28 

Plan  for  Collecting  Information 
from  Instructors,  V:  31-33 
Plan  for  Collecting  Information 
from  Students,  V:  29-30 
plan  (overall)  development,  V: 
11-33 

procedures,  V:  11-49 
Process  Evaluation  Plan,  V:  14- 
22 

product  evaluation,  V:  20 
Program  Evaluation  Review  Tech- 
nique (PERT),  V:  10 
Progress  Evaluation  Plan,  V:  11- 
14 

project  schedule,  V:  10 
report  (IN ER ) , V:  34,  49 
sample  summary  of  internal  eval- 
uation report,  V:  50 
Internal  Evaluation  Report  (INCR), 
V:  34,  49,  90,  93,  95,  105 
Interview(s)  for  data  collection, 
arrangements  for,  IV:  54 


Interview(s) , (continued) 

briefings  preceding,  I:  55-57 
criteria  for  interviewee  selection, 
I:  52-54,  59 

for  external  evaluation,  V:  74-75 
forms,  I:  49-51,  59,  134-136 
group,  I:  40 

guidelines  for  conducting,  I:  59- 
61 

individual,  on-site,  observation, 

I:  36-37,  41-46,  52-54 
schedule  for  conducting,  I:  58 


Job,  I:  8 

Job  analysis  , 1 : 1-112 

daily  schedule  for,  I:  58 
data  collection,  I:  34,  36-78 
data  revision,  I:  74-76 
documentation  for,  I:  83-85 
functions  of,  I:  2,  5 
interviews  for,  I:  36-37,  40-46, 
52-61 

job  analysts  (personnel),  I:  48- 
49 

of  equipment-oriented  jobs,  I:  81- 
83 

of  new  jobs,  I:  41 , 78-83 
of  non-equipment-orionted  jobs,  I: 
80-81 

outputs,  I:  33-34,  85 
overview  of,  I:  33-34 
procedures,  I:  34-83 
questionnaire  surveys  for,  I:  37- 
39,  44-46,  68-73 

task  inventory  validating,  1:  68 
task  inventory  verifying,  I:  66-67 
terms  defined,  I:  6-33 
Job  data  worksheets,  samples,  I:  51, 
63-65,  192-193,  210 
Job  Performance  Aids  (JPAs),  I:  18, 
114,  231 , 233-235,  245-249; 

III:  139,  252-260 
Job  performance  evaluation  data 
collection,  V:  71-73 
Job  Performance  Measures  (JPMs), 
in  analyzing  existing  courses,  I: 
216,  225-226 

in  developing  learning  objectives, 
II:  2,  6 

in  external  evaluation,  V:  70-73, 
80 

in  internal  evaluation, 


* 


V:  43 
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Job  Performance  Measures  (con- 
tinued) 

fn  selecting  instructional 
settings,  I:  232,  242 
in  system  revision,  V:  110-111 
in  testing,  II:  44 
Job  Performance  Measures,  con- 
struction of,  I:  156-212 
checklists  for,  I:  196-197,  211 
conditions,  I:  177-178 
cues,  I:  179 

cut-off  scores  for,  I:  200-201 
data  collection  for,  I:  45,  46 
documentation  for,  I:  191-193, 
209 

errors  in  measurement,  1:  201  - 
205 

outputs,  I:  209 
part-task  testing,  I:  185-187 
physical  fidelity,  I:  161-163, 
167 

predictive  validity,  I:  158-161, 
167 

procedures  for,  I:  165-209 
process  and  product  rating,  I: 
170-171,  195-196 
rationale  for,  I:  156-157 
rating  scales  for,  I:  197-200 
sample  list  of  validated  JPMs, 

I:  210 

sampling  plan,  I:  187-191 
scoring  procedures,  I:  191, 
194-204 

simulator  requirements  for,  I: 
163,  172-176 
standards,  I:  180-184 
testing  constraints,  I:  166- 
170 

tryout  procedure,  I:  207-208 
types  of  tasks  measured,  I: 
164-165 

validation  and  revision,  I: 
205-209 

Job  Performance  Test  (JPT),  I:  158 
Jury-of-Expert.s , I:  39-40,  41  , 67 


Knowledge  of  results  (K0R),  III: 
31  , 32,  33,  48,  51 


Learning  activities,  see  Learning 
events/activities,  specifying 


Learning  analysis,  II:  18-30 
attitude  category,  II:  28-30 
information  category,  II:  23-26 
mental  skills  category,  II:  19-23 
physical  skills  category,  II:  26- 
28 

Learning  categories  and  sub-categories, 
II:  16-17,  III:  11-17 
Learning  events/activities,  specifying, 
III:  1-105 

classification  of  learning 
objectives,  III:  10-17 
conditions  (active  vs.  passive), 

III:  4-5 

cues  or  prompts,  III:  6-8 
directions  for,  III:  24-26 
feedback,  III:  6,  9-10 
flowcharts  (algorithms),  III:  18, 

20,  71-104 

guidelines  for  eleven  sub- 
categories of  learning,  III: 
17-18,  35-70 

learning  categories  with  sub- 
categories, III:  11-17 
learning  guidelines  (general),  III: 
3-6 

outputs,  III:  26 
procedures.  III:  3-26 
purposes,  III:  1-3 
Learning  guidelines,  functions  of, 

III:  1-4,  208-213 
general,  III:  3-6,  10 
specific,  for  eleven  learning 
sub-categories,  III:  17-20, 

35-70 

sub-category  1:  rule-learning  and 
using,  III:  35-37,  71-73 
sub-category  2:  classifying,  III: 
38-40,  74-75 

sub-category  3:  identifying  symbols, 
III:  41-42,  76-77 
sub-category  4:  detecting,  III: 
43-45,  78-82 

sub-category  5:  making  decisions, 
III:  46-48,  83-86 

sub-category  6:  recalling  bodies  of 
knowledge.  III:  49-52,  87-90 
sub-category  7:  performing  gross 
motor  skills,  III:  53-55,  91-93 
sub-category  8:  steering  and  guid- 
ing, III:  56-57,  94-95 
sub-category  9:  positioning  move- 
ment, III:  58-62,  96-99 


i 
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Learning  guidelines  (continued) 
sub-category  10:  voice  conmuni- 
cating.  III:  63-65,  100-102 
sub-category  11:  attitude 
learning,  III:  66-70,  103- 
104 

Learning  objectives  (L0),  II:  1, 

4-5,  8-30,  see  al SO'  Objectives , 
common-factorT  II : 90-92 
testing,  II:  36,  38,  39.  42-45, 
49-50,  64 

Learning  Objective  Analysis  Work- 
sheet, II:  7 ',  III:  27,  109,  224- 
225 

Learning  objectives,  sequencing  and 
structuring,  see  Sequence  and 
structure  of  learning  objectives, 
determining 

Learning  Resource  Centers  (LRCs), 
III:  141 

Learning  steps  (LSs),  II:  4,  5,  36, 
38,  57 


Management  decisions,  see  Decisions, 
management 

Management  guidelines,  III:  124-137 
Management  plan,  see  Instructional 
management  plan  and  delivery 
system 

Management  plans,  student,  III:  185- 
196 

combination  plans,  III:  196 
computer-managed  instruction, 

III:  191-194 

conti ngency-managed  i nstruction , 
III:  187-189 

instructor-managed  instruction, 
III:  136-137 

media-managed  instruction,  III: 
194 

peer-managed  instruction.  III: 
190-191 

student  self-managed  instruction, 
III:  195 

Marginal  students,  guidelines  for 
managing,  III:  129-130 
Materials,  existing--review  of, 

see  Existing  materials,  review  of 
Matrices  (for  media  selection).  III: 
113,  115-118,  172-184 
Media,  comparative  costs,  III:  112 


Media  (continued) 

decision  matrices  for  selection 
of,  III:  172-184 
evaluation  of,  see  Existing 
materials,  review  of 
revision,  III:  206-208 
selection  (for  Instructional 
management  plan),  III:  106- 
124,  172-184 

Media-managed  instruction,  III:  194 

Media  specialist  (production  person- 
nel), III:  227 

Military  Occupational  Specialties 
(M0S),  I:  3,  8 

Misclassifications,  in  testing,  II: 
52-55 

Mnemonics,  III:  31,  36,  41,  50,  59 


Non-equipment-oriented  jobs,  I:  79- 
81 


Objectives,  development  of,  II:  1- 
34 

action  statement,  II:  5-12 
conditions  statement,  II:  5-6, 
9-10,  13-14 

criteria  for  Terminal  Learning 
Objective  and  Learning  Objec- 
tive statements,  II:  8-16 
inputs,  II:  1-2 

Job  Performance  Measures  in,  II: 
2 

learning  analysis  for  each  Term- 
inal Learning  Objective,  II: 
18-30 

learning  analysis:  attitude 
category,  II:  28-30 
learning  analysis:  information 
category,  II:  23-26 
learning  analysis:  mental  skills 
category,  II:  19-23 
learning  analysis:  physical 
skills  category,  II:  26-28 
learning  categories,  II:  16-17 
learning  objectives  (LOs),  II: 
4-5,  8-30 

learning  steps  (LSs),  II:  4-5 
outputs,  II:  30 
procedures,  II:  6-30 
standards  statement,  II:  5-6, 
9-10,  14-16 


Objectives,  development  of  (con- 
tinued) 

terminal  learning  objectives 
(TLOs)  preparation,  lit  2-5, 
6-16 

Peer-managed  instruction,  III:  190- 
191,  215 

Perceptual  set(s),  III:  46,  63 
Performance  Evaluation  Plan,  V:  24- 
28 

area  of  "entry  skills,"  V:  24- 
25 

area  of  "external  requirements, 

V:  23-24 

area  of  "tests,"  V:  25-26 
area  of  "time  required  to i com- 
plete instructional  units,1 
V:  27-28 

Performing  gross  motor  skills: 
learning  sub-category  7,  III: 
53-55,  91-93 

Photographer  (production  personnel), 
III:  228 

Physical  fidelity,  of  dob  Perform- 
ance Measures,  I:  161-163,  lb/ 
Platform  lectures,  III:  248 
Positioning  movement,  and  recalling 
procedures:  learning  sub-category 
9,  III:  58-62,  96-99 
Post-feedback  delay,  III:  61 
Posttest,  V : 109-110 
Predictive  validity  of  dob  Per- 
formance Measures  , 1:  158-161, 

167  . „T 

Predifferenti ation  of  stimuli.  III: 
38,  41 , 49 
Pretest,  II:  73-75 
Print  specialist  (production  per- 
sonnel ) , III:  228 
Process  Evaluation  Plan,  V.  14-22 
Process  rating,  I:  196 
Product  rating,  I:  195 
Program  Evaluation  Review  Tech- 
nique (PERT),  V:  10 
Programmed  instruction,  III:  246- 
247 

Progress  Evaluation  Plan,  V:  11-14 


Questionnaire  survey,  administra- 
tion of,  I:  72-73,  106-112 


Questionnaire  survey  (continued) 
closed  form,  1:  38-39 
for  external  evaluation  data 
collection,  V:  73-74 
for  job  analysis,  I:  37-39,  44- 
46 

for  rating  tasks  for  training, 
I:  133-136 
open  form,  I:  38-39 
preparation  of,  I:  68-71,  134- 
136 

sample  selection  for,  I:  71-72, 
133-134 


Random  grouping,  III:  125 
Rank-order  testing,  II:  52-55 
Ratings  (Navy),  I:  3,  8 
Rating  scales  (for  dob  Performance 
Measures),  description  of,  I: 
197-198 

graphic,  I:  198-200 
numerical,  I:  197 
Recalling  bodies  of  knowledge: 
learning  sub-category  6,  III: 
49-52,  87-90 

Reinforcement;  reinforcer,  III: 
187-189 

Rasident  Schools  (R$)>  I*  238- 
239,  259;  III:  139-141 
Response  biases,  III:  46 
Revision  of  system,  see  System 
revision 

Revision  olan  format,  V:  113 
Rule-learning  and  using:  learning 
sub-category  1,  HI:  3o-37,  71-7. 


Sampling  plan  for  Job  Performance 
Measures,  I:  187-191 
Self-pacing  instructional  plan,  III: 
124 , 125-126 

Self-Teaching  Exportable  Packages 
(STEPs),  I:  231,  235-237,  250- 
255;  III:  140-142,  248-249 
Sequence  and  structure  of  learning 
objectives,  rieterming,  II:  79-9b 
conmon-factor  learning  objectives, 
II:  90-92 

determining  relationships,  In 
81-82,  83 

grouping,  II:  92-94 
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Sequence  and  structure  of  learning 
objectives,  determining  (contin- 
ued) 

outputs,  II:  94 
procedures,  II:  81-93 
purposes  of,  II:  79-80,  88-89 
with  dependent  relationships, 

II:  82-87 

with  independent  relationships, 
II:  90 

with  su^  ortive  relationships, 
II:  .7-88 

Sotting,  see  Instructional  set- 
ting, seTaction  of 
Shaping,  III:  55,  57 
Simulator  requirements  in  Job 
Performance  Measures,  I:  163, 
172-176 

Slide-tape  production,  III:  239- 
244 

Soft  data , V : 4 

Source  materials,  of  job  analysis 
data  and  job  information,  I: 
97-98 

of  training  courses  and  in- 
structional materials,  I: 
95-97 

Staff  training,  instructional, 

IV:  7-12 

Standards,  Job  Performance 
Measures,  I:  180-184 
task,  I:  28-33 
terminal  learning  objective 
test,  II:  47-50 
training,  II:  47-50 
State-of-the-art,  III:  120 
Steering  and  guiding:  learning 
sub-category  8,  III:  56-57, 
94-95 

Stimulus  criteria  (media),  III: 
107-108 

Storyboards,  III:  234,  239-242, 
245-246 

Student  management  plans,  see 
Management  plans,  student  and 
Instructional  management  plan 
Student  self-managed  instruction, 
III:  195,  215 
Students'  Guide,  III:  269 
Students ' manual  , IV  : 6 
Subject  Matter  Experts  (SMEs), 

I:  79,  81-82,  133 


Supplementary  instruction,  III:  249- 
250 

System  Master  Plan  (SMP),  III:  105, 
107,  138-142;  V:  10,  90-91 
System  revision,  V:  87-120 

based  on  changes  in  doctrine  or 
content  of  DOS,  V:  94 
based  on  efficiency  of  instruc- 
tion, V:  96-103 
based  on  External  Evaluation 
Report,  V:  95-96 
based  on  Internal  Evaluation 
Report,  V:  95 
description  of,  V:  87-89 
follow-up  activity,  V:  115 
for  improvement  of  instructional 
effectiveness,  V:  109-112 
for  operating  instruction,  V: 
105-106 

for  time  reduction,  V:  106-109 
guidelines  for  determining  revi- 
sion needs,  V:  103-112 
outputs,  V:  115-116 
preparation  of  revision  plan, 

V:  112-114 

priority  ratings  for,  V:  92 
procedures  for,  V:  93-115 
purposes  of,  V:  87-90,  93 
sources  for,  V:  90-91 


Task(s) , I:  12 

checklist  task  inventory,  I: 
152-154 

clustering  of,  I:  244-245 
conditions  statement  for,  I:  24- 
26,  89-93 

delay  tolerance,  I:  123-125,  246 
guidelines  for  diagramming,  I: 

24 

inventory,  verifying  and  validat- 
ing, I:  66-68 

Job  Performance  Measures  for,  I: 
157-211 

multiple,  I:  164-165 
statements,  I:  13-17,  61 
unitary,  I:  164 

validated  task  list,  sample,  I: 
33,  86 

Task  selection,  I:  113-155 

criteria  for,  I:  118-133,  152- 
1 54 
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Task  selection  (continued) 

data  assessing  and  decision- 
making  for,  I:  139-148 
data  collection  for,  I:  134-136 
data  consolidation  for,  I:  136- 
139 

management  constraints,  I:  143- 
144 

outputs,  I:  117,  148-149 
procedures,  I:  117-148 
rationale  for,  1:  115-117 
survey  sources,  1:  133-134 
Television  producer  (production 
personnel),  III:  229 
Television  program  production.  III: 
244 

Terminal  learning  objectives  (TLOs), 
II:  1-31 

action  statement  O'*7,  II:  5-12 
conditions  statement  of,  II:  5-6, 
9-10,  13-14 

guidelines  for  learner,  III:  4 
learning  analysis  for,  II:  18-30 
learning  categories,  II:  16-17 
standards  statement  of,  II:  5-6, 
9-10,  14-16 

Test  development,  II:  35-61 
attitudes  testing,  II:  45-46 
information  testing,  II:  42-44 
inputs,  II:  7 

mental  skills  testing,  II:  39-42 
inisclassifications,  II:  50-52 
outputs  , II:  37  , 60 
physical  skills  testing,  II:  44-45 
procedures,  II:  38-60 
purposes,  II:  35 
rank-order  testing,  II:  52-55 
scoring,  II:  55-56 
standards,  II:  47-50 
types  of  tests,  II:  35,  38-39 
Testing  constraints  in  Job  Perform- 
ance Measures  , I:  166-177 
Trainer  Appraisal  Kit  (TAK),  IV: 

7-12 

Trainer  Development  Proyram  (TRADEP), 
IV:  9 

Training  Extension  Course  (TEC), 

Army,  III:  141-142 
Training  programs,  selection  of 
tasks  for,  I:  114-133,  139-148 
Training  task  categories,  see 
Learning  categories 


Trial  report  (small  group),  example, 
III:  340-345 


Voice  communicating:  learning  sub- 
category 10,  III:  63-65,  100- 
102 


Writer  (production  personnel),  III: 
229 


